Methods and systems for obtaining, aggregating, and analyzing vision data to assess a person&#39;s vision performance

ABSTRACT

The present specification describes methods and systems for modifying a media, such as Virtual Reality, Augmented Reality, or Mixed Reality (VR/AR/MxR) media based on a vision profile and a target application. In embodiments of the specification, a Sensory Data Exchange (SDE) is created that enables identification of various vision profiles for users and user groups. The SDE may be utilized to modify one or more media in accordance with each type of user and/or user group.

CROSS-REFERENCE

The present application is a continuation application of U.S. patentapplication Ser. No. 15/482,560, entitled “Methods and Systems forObtaining, Aggregating, and Analyzing Vision Data to Assess a Person'sVision Performance” and filed on Apr. 7, 2017, which relies on, forpriority, the following United States Provisional patent applications:

U.S. Provisional Patent Application No. 62/425,736, entitled “Methodsand Systems for Gathering Visual Performance Data and Modifying MediaBased on the Visual Performance Data” and filed on Nov. 23, 2016;

U.S. Provisional Patent Application No. 62/381,784, of the same titleand filed on Aug. 31, 2016;

U.S. Provisional Patent Application No. 62/363,074, entitled “Systemsand Methods for Creating Virtual Content Representations Via A SensoryData Exchange Platform” and filed on Jul. 15, 2016;

U.S. Provisional Patent Application No. 62/359,796, entitled “VirtualContent Representations” and filed on Jul. 8, 2016;

U.S. Provisional Patent Application No. 62/322,741, of the same titleand filed on Apr. 14, 2016; and

U.S. Provisional Patent Application No. 62/319,825, of the same titleand filed on Apr. 8, 2016.

All of the aforementioned applications are incorporated herein byreference in their entirety.

FIELD

The present specification relates generally to vision care and morespecifically to methods and systems for obtaining, aggregating, andanalysing vision data to assess a person's vision performance.

BACKGROUND

In recent years, the advent of various visual experiences, includingVirtual Reality (VR) environments, Augmented Reality (AR), and MixedReality (MxR) applications through various mediums, such as tabletcomputers and mobile phones, have placed a greater strain on the visionof users. Reliable measurements of the strain on vision requires anunderstanding of numerous psychometrics and how various visual fieldparameters affect those psychometrics, and how those vision fieldparameters can be modified in order to avoid certain vision problems.

In turn, this requires an understanding of the interoperability,connectivity, and modularity of multiple sensory interfaces with thebrain, with many being closed-looped.

Current measures and rating systems for AR/VR are qualitative in nature.Further, clinical testing interfaces include EEG, MRI, EOG, MEG, fMRI,ultrasound, and microwaves. Traditional industry standards for measuringField of View include tests such as The Amsler grid, the Humphrey VisualField Analyzer, Frequency-Doubling technology, the Tangent Screen Exam,the Goldmann Method, and the Octopus perimeter. For Accuracy,compensatory tracking, Jenson Box, and Hick's Law tests/standards aretypically used. Industry standard tests for multi-tracking includeauditory serial addition, the Posner Cueing Task, and the D2 Test ofAttention. For Endurance, typical industry standard tests include VisualField Perimetry (maintaining fixation) and Optical Coherence Tomography(OCT) Tests. Industry standard Detection tests include Ishihara test(color vision/color plates), Farnsworth-Munsell 100 hue test, PelliRobson Contrast Sensitivity Chart, Vistech Contrast test, SnellenCharts, ETDRS, and Tumbling Cs.

While these traditional industry standards and clinical standards existfor sight testing, there is still a need for a comprehensive visualperformance index or assessment that integrates multiple, disparatemeasures into a single aggregated measurement. What is also needed is asoftware interface that provides an aggregate quantification of multipledata points. What is also needed is a method and system for monitoringeye health and identifying changes to vision over time.

SUMMARY

The present specification is directed toward a method of assessing avision performance of a patient using a computing device programmed toexecute a plurality of programmatic instructions, comprising presenting,via the computing device, a first set of visual and/or auditory stimuli;monitoring a first plurality of reactions of the patient using at leastone of the computing device and a separate hardware device; presenting,via the computing device, a second set of visual and/or auditorystimuli; monitoring a second plurality of reactions of the patient usingat least one of the computing device and a separate hardware device; andbased upon said first plurality of reactions and second plurality ofreactions, determining quantitative values representative of thepatient's field of view, visual acuity, ability of the patient to trackmultiple stimuli, visual endurance and visual detection.

Optionally, the method further comprises generating a single visionperformance value representative of an aggregation of the field of view,the visual acuity, the ability of the patient to track multiple stimuli,the visual endurance and the visual detection. Optionally, the firstplurality of reactions comprises at least one of rapid scanning data,saccadic movement data, blink rate data, fixation data, pupillarydiameter data, and palpebral fissure distance data. Optionally, thesecond plurality of reactions comprises at least one of rapid scanningdata, saccadic movement data, fixation data, blink rate data, pupillarydiameter data, speed of head movement data, direction of head movementdata, heart rate data, motor reaction time data, smooth pursuit data,palpebral fissure distance data, degree and rate of brain wave activitydata, and degree of convergence data.

Optionally, the hardware device comprises at least one of a cameraconfigured to acquire eye movement data, a sensor configured to detect arate and/or direction of head movement, a sensor configured to detect aheart rate, and an EEG sensor to detect brain waves. Optionally, thequantitative values representative of the patient's field of viewcomprises data representative of a quality of the patient's centralvision and data representative of a quality of the patient's peripheralvision. Optionally, the quantitative values representative of thepatient's visual acuity comprises data representative of a quality ofthe patient's reaction time to said first set of visual and/or auditorystimuli. Optionally, the quantitative values representative of thepatient's visual acuity comprises data representative of a quality ofthe patient's precise targeting of said first set of visual stimuli andwherein said quality of the patient's precise targeting of said firstset of visual stimuli is based on a position of the patient's physicalresponse relative to a position of the first set of visual stimuli.

Optionally, the quantitative values representative of the patient'sability of the patient to track multiple stimuli comprises datarepresentative of a quality of the patient's ability to simultaneoustrack multiple elements in the second set of visual stimuli. Optionally,the quantitative values representative of the patient's visual endurancecomprises data representative of a decrease in the patient's reactiontime over a duration of presenting the first set of visual and/orauditory stimuli. Optionally, the quantitative values representative ofthe patient's visual endurance comprises data representative of animprovement in the patient's reaction time over a duration of presentingthe second set of visual and/or auditory stimuli after a rest period.Optionally, the quantitative values representative of the patient'svisual detection comprises data representative of to what extent thepatient sees the first set of visual stimuli. Optionally, thequantitative values representative of the patient's visual detectioncomprises data representative of to what extent the patient candiscriminate between similarly colored, contrasted, or shaped objects inthe first set of visual stimuli.

In another embodiment, the present specification is directed to a methodof assessing a vision performance of a patient using a computing deviceprogrammed to execute a plurality of programmatic instructions,comprising presenting, via a display on the computing device, a firstset of visual stimuli, wherein the first set of visual stimuli comprisesa first plurality of visual elements that move from a peripheral visionof the patient to a central vision of the patient; monitoring a firstplurality of reactions of the patient using at least one of thecomputing device and a separate hardware device; presenting, via adisplay on the computing device, a second set of visual stimuli, whereinthe second set of visual stimuli comprises a second plurality of visualelements that appear and disappear upon the patient physically touchingsaid second plurality of visual elements; monitoring a second pluralityof reactions of the patient using at least one of the computing deviceand said separate hardware device; and based upon said first pluralityof reactions and second plurality of reactions, determining quantitativevalues representative of the patient's field of view, visual acuity,ability of the patient to track multiple stimuli, visual endurance andvisual detection.

Optionally, at least a portion of the first plurality of visual elementshave sizes that decrease over time. Optionally, at least a portion ofthe first plurality of visual elements have a speed of movement thatincreases over time. Optionally, over time, more of the first pluralityof visual elements simultaneously appear on said computing device.Optionally, a third plurality of visual elements appear concurrent withsaid second plurality of visual elements, wherein the third plurality ofvisual elements appear different than the second plurality of visualelements, and wherein, if the patient physically touches any of saidthird plurality of visual elements, the quantitative valuerepresentative of the patient's visual acuity is decreased.

Optionally, the method further comprises presenting, via a display onthe computing device, a third set of visual stimuli, wherein the thirdset of visual stimuli comprises a fourth plurality of visual elements;monitoring a third plurality of reactions of the patient using at leastone of the computing device and said separate hardware device; and basedupon said first plurality of reactions, second plurality of reactions,and third plurality of reactions determining quantitative valuesrepresentative of the patient's field of view, visual acuity, ability ofthe patient to track multiple stimuli, visual endurance and visualdetection. Optionally, the patient is instructed to identify one of thefourth plurality of visual elements having a specific combination ofcolor, contrast, and/or shape.

It should be appreciated that while the method is described above ashaving a particular order of presenting visual stimuli, the presentinvention is directed toward any order of presenting the visual elementsand corresponding monitoring for specific patient vision quantitativevalues. For example, optionally, at least a portion of the secondplurality of visual elements have sizes that decrease over time.Optionally, at least a portion of the second plurality of visualelements have a speed of movement that increases over time. Optionally,over time, more of the second plurality of visual elementssimultaneously appear on said computing device. Optionally, a thirdplurality of visual elements appear concurrent with said first pluralityof visual elements, wherein the third plurality of visual elementsappear different than the first plurality of visual elements, andwherein, if the patient physically touches any of said third pluralityof visual elements, instead of the first plurality of visual elements,the quantitative value representative of the patient's visual acuity isdecreased.

The aforementioned and other embodiments of the present shall bedescribed in greater depth in the drawings and detailed descriptionprovided below.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features and advantages of the present specificationwill be appreciated, as they become better understood by reference tothe following detailed description when considered in connection withthe accompanying drawings, wherein:

FIG. 1 shows a block diagram illustrating user interaction with anexemplary Sensory Data Exchange Platform (SDEP), in accordance with anembodiment of the present specification;

FIG. 2A is a block diagram illustrating processing of a sensor datastream before it reaches a query processor, in accordance with anembodiment of the present specification;

FIG. 2B is an exemplary outline of a data analysis chain;

FIG. 3 illustrates an overview of sources of digital data, in accordancewith an embodiment of the present specification;

FIG. 4 illustrates characteristic metrics for visual data, in accordancewith an embodiment of the present specification;

FIG. 5 provides a graphical presentation of color pair confusioncomponents, in accordance with an embodiment of the presentspecification;

FIG. 6 shows a graph illustrating how luminance may be found for a givenchromaticity that falls on the top surface of the display gamutprojected into 3D chromoluminance space;

FIG. 7 illustrates characteristic metrics for auditory information, inaccordance with an embodiment of the present specification;

FIG. 8 illustrates characteristic metrics for eye tracking, inaccordance with an exemplary embodiment of the present specification;

FIG. 9 illustrates characteristic metrics for manual input, inaccordance with an embodiment of the present specification;

FIG. 10 illustrates characteristic metrics for head tracking, inaccordance with an embodiment of the present specification;

FIG. 11 illustrates characteristic metrics for electrophysiological andautonomic monitoring data, in accordance with an embodiment of thepresent specification;

FIG. 12A illustrates an exemplary process of image analysis of buildingcurated data, in accordance with an embodiment of the presentspecification;

FIG. 12B illustrates an exemplary process of image analysis of buildingcurated data, in accordance with an embodiment of the presentspecification;

FIG. 12C illustrates an exemplary process of image analysis of buildingcurated data, in accordance with an embodiment of the presentspecification;

FIG. 12D illustrates an exemplary process of image analysis of buildingcurated data, in accordance with an embodiment of the presentspecification;

FIG. 13A illustrates pupil position and size and gaze position overtime;

FIG. 13B illustrates pupil position and size and gaze position overtime;

FIG. 14 provides a table containing a list of exemplary metrics forafferent and efferent sources, in accordance with some embodiments ofthe present specification;

FIG. 15 is an exemplary flow chart illustrating an overview of the flowof data from a software application to the SDEP;

FIG. 16 is an exemplary outline of a pre-processing portion of a processflow, in accordance with an embodiment of the present specification;

FIG. 17 is an exemplary outline of a python scripting portion of theanalysis chain;

FIG. 18 illustrates an exemplary environment for implementing a centralsystem that utilizes the SDEP to process psychometric functions and tomodel visual behavior and perception based on biomimicry of userinteraction;

FIG. 19 illustrates screenshots of empty and error screens that mayappear through the sight kit application, in accordance with anembodiment of the present specification;

FIG. 20A illustrates a screenshot of splash screen that may appearthrough the sight kit application, in accordance with an embodiment ofthe present specification;

FIG. 20B illustrates a screenshot of home screen that may appear throughthe sight kit application, in accordance with an embodiment of thepresent specification;

FIG. 20C illustrates a series of screenshots of the login (registration)process including an exemplary registration by a user named Jon Snow′that may appear through the sight kit application, in accordance with anembodiment of the present specification;

FIG. 20D illustrates a screenshot of a screen with terms and conditionsthat may appear through the sight kit application, in accordance with anembodiment of the present specification;

FIG. 20E illustrates a series of screenshots that may appear through thesight kit application in case a user forget their login information, inaccordance with an embodiment of the present specification;

FIG. 21A illustrates a series of screenshots of screens that prompt auser with demographic questions that may appear through the sight kitapplication, in accordance with an embodiment of the presentspecification;

FIG. 21B illustrates a further series of screenshots of screens thatprompt a user with demographic questions that may appear through thesight kit application, in accordance with an embodiment of the presentspecification;

FIG. 21C illustrates still further series of screenshots of screens thatprompt a user with demographic questions that may appear through thesight kit application, in accordance with an embodiment of the presentspecification;

FIG. 22 illustrates a series of screenshots of screens that present auser with an initial VPI report that may appear through the sight kitapplication, in accordance with an embodiment of the presentspecification;

FIG. 23 illustrates screenshots of different screens that may appear atseparate times, prompting a user to select a game to play that mayappear through the sight kit application, in accordance with anembodiment of the present specification;

FIG. 24A illustrates a screenshot of Pop the Balloons Round 1instructions, which may be presented through the sight kit applicationin accordance with an embodiment of the present specification;

FIG. 24B illustrates a screenshot of Pop the Balloons Round 1 game,which may be presented through the sight kit application in accordancewith an embodiment of the present specification;

FIG. 24C illustrates a screenshot of Pop the Balloons Round 2instructions, which may be presented through the sight kit applicationin accordance with an embodiment of the present specification;

FIG. 24D illustrates a screenshot of Pop the Balloons Round 2 game,which may be presented through the sight kit application in accordancewith an embodiment of the present specification;

FIG. 24E illustrates a screenshot of Pop the Balloons Round 3instructions, which may be presented through the sight kit applicationin accordance with an embodiment of the present specification;

FIG. 24F illustrates a screenshot of Pop the Balloons Round 3 game,which may be presented through the sight kit application in accordancewith an embodiment of the present specification;

FIG. 25A illustrates a series of screenshots of Picture Perfect Round 1game, which may be presented through the sight kit application inaccordance with an embodiment of the present specification;

FIG. 25B illustrates a series of screenshots of Picture Perfect Round 1game, which may be presented through the sight kit application inaccordance with an embodiment of the present specification;

FIG. 25C illustrates a series of screenshots of Picture Perfect Round 2game, which may be presented through the sight kit application inaccordance with an embodiment of the present specification;

FIG. 25D illustrates a series of screenshots of Picture Perfect Round 2game, which may be presented through the sight kit application inaccordance with an embodiment of the present specification;

FIG. 25E illustrates a series of screenshots of Picture Perfect Round 2game, which may be presented through the sight kit application inaccordance with an embodiment of the present specification;

FIG. 25F illustrates a screenshot of an exemplary after game report fora user, which may be presented through the sight kit application inaccordance with an embodiment of the present specification;

FIG. 26A illustrates a similar set of screenshots for ‘Shape Remix’game, its instructions, and after game report, which may be presentedthrough the sight kit application in accordance with an embodiment ofthe present specification;

FIG. 26B illustrates a similar set of screenshots for ‘Shape Remix’game, its instructions, and after game report, which may be presentedthrough the sight kit application in accordance with an embodiment ofthe present specification;

FIG. 26C illustrates a similar set of screenshots for ‘Shape Remix’game, its instructions, and after game report, which may be presentedthrough the sight kit application in accordance with an embodiment ofthe present specification.

FIG. 27 illustrates screenshots of VPI game reports after playingdifferent games that may appear through the sight kit application, inaccordance with an embodiment of the present specification;

FIG. 28 illustrates some screenshots that may appear based on the user'sVPI report, where the screens suggest doctors and/or eye-carepractitioners, in accordance with an embodiment of the presentspecification;

FIG. 29 illustrates some screenshots of the screens that present auser's profile that may appear through the sight kit application, inaccordance with an embodiment of the present specification;

FIG. 30A illustrates some screenshots of the VPI breakdown that mayappear through the sight kit application, in accordance with anembodiment of the present specification;

FIG. 30B illustrates some screenshots of the VPI breakdown that providedetails about each FAMED parameter, through the sight kit application inaccordance with an embodiment of the present specification;

FIG. 30C illustrates some screenshots of the VPI breakdown that providedetails of parameters within each FAMED parameter, through the sight kitapplication in accordance with an embodiment of the presentspecification;

FIG. 30D illustrates some screenshots of the VPI breakdown that providefurther details of parameters within each FAMED parameter, through thesight kit application in accordance with an embodiment of the presentspecification;

FIG. 31 illustrates screenshots for ‘Settings’ and related optionswithin ‘Settings’, which may be presented through the sight kitapplication in accordance with an embodiment of the presentspecification; and

FIG. 32 is a table showing exemplary experiences of different VPIparameters from the different games and rounds.

DETAILED DESCRIPTION

In one embodiment, the present specification describes methods, systemsand software that are provided to vision service providers in order togather more detailed data about the function and anatomy of human eyesin response to various stimuli.

In one embodiment, a Sensory Data Exchange Platform (SDEP) is provided,wherein the SDEP may enable developers of games, particularly mobileapplications or other media and/or software, to optimize the media for auser and/or a group of users. In embodiments, the SDEP, or at least aportion thereof, is embodied in a software application that is presentedto an end-user through one or more electronic media devices includingcomputers, portable computing devices, mobile devices, or any otherdevice that is capable of presenting virtual reality (VR), augmentedreality (AR), and/or mixed reality MxR media.

In an embodiment, a user interacts with a software program embodying atleast a portion of the SDEP in a manner that enables the software tocollect user data and provided it to the SDEP. In an embodiment, theuser may interact directly or indirectly with a SDEP to facilitate datacollection. In an embodiment, the SDEP is a dynamic, two-way dataexchange platform with a plurality of sensory and biometric data inputs,a plurality of programmatic instructions for analyzing the sensory andbiometric data, and a plurality of outputs for the delivery of anintegrated visual assessment.

In some embodiments, the SDEP outputs as a general collective output a“visual data profile” or a “vision performance index” (VPI). In someembodiments, the SDEP outputs as a general collective output a visionperformance persona. The visual data profile or vision performance indexmay be used to optimize media presentations of advertising, gaming, orcontent in a VR/AR/M×R system. In embodiments, the platform of thepresent specification is capable of taking in a number of other datasets that may enhance the understanding of a person's lifestyle andhabits. In addition, machine learning, computer vision, and deeplearning techniques are employed to help monitor and predict healthoutcomes through the analysis of an individual's data. In embodiments,the vision performance index is employed as a tool for measuring visionfunction. In embodiments, the vision performance index may be generatedbased upon any plurality or combination of data described throughoutthis specification and is not limited to the examples presented herein.

In an embodiment, the SDEP is used via an operating system executed onhardware (such as mobile, computer or Head Mounted Display (HMD)). Inanother embodiment, the SDEP is used by one or more content developers.In one embodiment, both hardware and content developers use the SDEP.The SDEP may enable collection of data related to how the user isinterfacing with the content presented, what aspects of the content theyare most engaged with and how engaged they are. Data collected throughthe SDEP may be processed to create a profile for the user and or groupsof users with similar demographics. The content may be represented, fora particular profile, in a way that conforms to the hardwarecapabilities of the VR/AR/M×R system in a manner to optimize experienceof that user and other users with a similar profile.

The present specification is directed towards multiple embodiments. Thefollowing disclosure is provided in order to enable a person havingordinary skill in the art to practice the invention. Language used inthis specification should not be interpreted as a general disavowal ofany one specific embodiment or used to limit the claims beyond themeaning of the terms used therein. The general principles defined hereinmay be applied to other embodiments and applications without departingfrom the spirit and scope of the invention. Also, the terminology andphraseology used is for the purpose of describing exemplary embodimentsand should not be considered limiting. Thus, the present invention is tobe accorded the widest scope encompassing numerous alternatives,modifications and equivalents consistent with the principles andfeatures disclosed. For purpose of clarity, details relating totechnical material that is known in the technical fields related to theinvention have not been described in detail so as not to unnecessarilyobscure the present invention.

The term “and/or” means one or all of the listed elements or acombination of any two or more of the listed elements.

The terms “comprises” and variations thereof do not have a limitingmeaning where these terms appear in the description and claims.

Unless otherwise specified, “a,” “an,” “the,” “one or more,” and “atleast one” are used interchangeably and mean one or more than one.

For any method disclosed herein that includes discrete steps, the stepsmay be conducted in any feasible order. And, as appropriate, anycombination of two or more steps may be conducted simultaneously.

Also herein, the recitations of numerical ranges by endpoints includeall whole or fractional numbers subsumed within that range (e.g., 1 to 5includes 1, 1.5, 2, 2.75, 3, 3.80, 4, 5, etc.). Unless otherwiseindicated, all numbers expressing quantities of components, molecularweights, and so forth used in the specification and claims are to beunderstood as being modified in all instances by the term “about.”Accordingly, unless otherwise indicated to the contrary, the numericalparameters set forth in the specification and claims are approximationsthat may vary depending upon the desired properties sought to beobtained by the present invention. At the very least, and not as anattempt to limit the doctrine of equivalents to the scope of the claims,each numerical parameter should at least be construed in light of thenumber of reported significant digits and by applying ordinary roundingtechniques.

Notwithstanding that the numerical ranges and parameters setting forththe broad scope of the invention are approximations, the numericalvalues set forth in the specific examples are reported as precisely aspossible. All numerical values, however, inherently contain a rangenecessarily resulting from the standard deviation found in theirrespective testing measurements.

It should be noted herein that any feature or component described inassociation with a specific embodiment may be used and implemented withany other embodiment unless clearly indicated otherwise.

It should be further appreciated that all the afferent data presentedherein and efferent data collected are performed using a hardwaredevice, such as a mobile phone, laptop, tablet computer, or specialtyhardware device, executing a plurality of programmatic instructionsexpressly designed to present, track, and monitor afferent data and tomonitor, measure, and track efferent data, as further discussed below.

General Definitions

The term “Virtual Reality” or “VR” is used throughout thisspecification, and, in embodiments, refers to immersivecomputer-simulated reality, or the computer-generated simulation of athree-dimensional image or environment that can be interacted with in aseemingly real or physical way by a person using special electronicequipment, such as a helmet with a screen inside and/or gloves fittedwith sensors.

In embodiments, Augmented Reality (AR), also used along with VRthroughout this specification, is a technology that superimposes acomputer-generated image on a user's view of the real world, thusproviding a composite view. In embodiments, a common helmet-like deviceis the HMD, which is a display device, worn on the head or as part ofthe helmet, that has a small display optic in front of one (monocularHMD) or each eye (binocular HMD). In embodiments, the SDEP is acloud-based service that any party can access in order to improve orotherwise modify a visually presented product or service.

Further, in embodiments, Mixed Reality (M×R), is also used with VR andAR throughout this specification. M×R, also referred to as hybridreality, is the merging of VR and/or AR environments with the realenvironment to produce new levels of visual-experiences where physicaland digital objects co-exist and interact in real time.

In embodiments, VR, AR, and M×R devices could include one or more ofelectronic media devices, computing devices, portable computing devicesincluding mobile phones, laptops, personal digital assistants (PDAs), orany other electronic device that can support VR, AR, or M×R media. Itshould be noted herein that while the present specification is disclosedin the context of Virtual Reality, any and all of the systems andmethods described below may also be employed in an Augmented Realityenvironment as well as Mixed Reality environments. So, where a VirtualReality (VR) system is described, it should be understood by those ofordinary skill in the art that the same concepts may apply to anAugmented Reality (AR) and a Mixed Reality (M×R) system.

Eye-Tracking Definitions

In terms of performance, several eye tracking measures are put into thecontext of Vision Performance Index (VPI) components, which are definedand described in detail in subsequent section of the specification.Blink rate and vergence measures can feed into measures of fatigue andrecovery. Gaze and, more specifically, fixation positions can be used toestimate reaction and targeting measures. Continuous error rates duringpursuit eye movements can also become targeting measures.

In embodiments, the vision performance index is employed as a tool formeasuring vision function. In embodiments, the vision performance indexmay be generated based upon any plurality or combination of datadescribed throughout this specification and is not limited to theexamples presented herein.

Various examples of physical measures for eye tracking may be availablewith desired standard units, expected ranges for measured values and/or,where applicable, thresholds for various states or categories based onthose measures. Some references are provided through sections thatdiscuss various components and subcomponents of eye tracking.

The following terms are associated with eye-tracking measures as madefrom a combination of video recording and image processing techniques;expert human scoring; and/or from electrooculography (EOG) recording.Video eye tracking (VET) techniques may use explicit algorithmicanalysis and/or machine learning to estimate proportional eyelidopening/closure, pupil size, pupil position (relative to the face) andgaze direction independently for each eye. EOG recording may be used toestimate eyelid and eye motion and, with limited precision, eye gazedirection. Both recording modalities may sample at rates of tens tothousands of times per second and allow for analysis of position,velocity, direction, and acceleration for the various measures.Comparison between the two eyes allows for measures of vergence which inturn allows for a three-dimensional (3D) gaze direction to be estimated.

Palpebral Fissure refers to the opening of the eyelids. While typicallyabout 30 millimeters (mm) wide by 10 mm tall, most measurements can berelative to baseline distances measured on video. Of particular interestis the height (interpalpebral fissure height) as it relates to thefollowing terms:

Percent Open (p_(eye open)) refers to how open the left(p_(left eye open)), right (p_(right eye open)), or both(p_(both eyes open)) eyes are, relative to the maximum open distance andtypically measured over a predefined period of time.

Proportion Open (P_(eyes open)) refers to the proportion of time theeyes are open over a span of time (for example, during a session(P_(eyes open|session))). The threshold for ‘open’ may be variable (forexample, P_(eyes open)(where p_(both eyes open)≥25%)).

Blink can be defined as a complete closure of both eyes(p_(both eyes open)=0%) for between roughly 10 to 400 milliseconds (ms),with a specific measured blink closure time being based on differencesamong users and the eye tracking method.

Blink Rate (Frequency) (f_(blink)) refers to the average number ofblinks per second (s⁻¹ or Hz) measured for all blinks and/or blinks overa period of time (e.g. f_(blink|target present)). The blink rate may bereferred to as a rate of change of the blink rate or a ratio of partialblinks to full blinks.

Blink Count Number (N_blink) refers to the number of blinks measured forall blinks and/or blinks over a period of time (e.g. N_(blink|targetpresent)).

Pupil Size (S_pupil) refers to the size of the pupil, typically thediameter in millimeters (mm).

Pupil Position (

[x,y]

_pupil) refers to the position of the left (

[x,y]

_(left pupil)) or right (

[x, y]

_(right pupil)) pupil within the fixed reference frame of the face,typically as a function of time. The pupil position definition includes,and is dependent upon, an initial pupil position and a final pupilposition.

Gaze Direction (

[θ,ϕ]

_gaze) refers to the direction in 3D polar coordinates of left (

[θ,ϕ]

_(left gaze)) or right (

[θ,ϕ]

_(right gaze)) eye gaze relative to the face, typically as a function oftime. This is a measure of where the eyes are facing without regard towhat the eyes see. It may be further classified as relevant orirrelevant depending on a task or a target.

Gaze Position (

[x, y, z]

_gaze or

[r, θ, φ]

_gaze) refers to the position (or destination) of gaze in theenvironment in Cartesian or spherical 3D coordinates, typically as afunction of time. The reference frame may be with respect to the user,device or some other point in space, but most commonly the origin of acoordinate space will be the user's eyes (one or the other or a pointhalfway between). The gaze position definition includes, and isdependent upon, an initial gaze position and a final gaze position.

Vergence is derived from estimated gaze direction and may be quantifiedas the difference in angle of the two eyes (positive differences beingdivergence and negative being convergence). When derived from gazeposition, vergence contributes to and may be quantified as the distanceof the gaze position from the eyes/face. Convergence and divergence mayeach be defined by their duration and rate of change.

Fixation Position ([x, y,]_(fixation) or [r, θ, φ] fixation) is theposition of a fixation in Cartesian or spherical 3D space measured asthe estimated position of the user's gaze at a point in time. Thefixation position definition includes, and is dependent upon, an initialfixation position and a final fixation position.

Fixation Duration (D_(fixation)) is the duration of a fixation (i.e. thetime span between when the gaze of the eye arrives at a fixed positionand when it leaves), typically measured in milliseconds or seconds (s).The average duration is denoted with a bar D _(fixation) and mayrepresent all fixations, fixations over a period of time (e.g._D_(fixation|target present)) and/or fixations within a particularregion (e.g. _D_(fixation|display center)). The fixation durationdefinition includes, and is dependent upon, a rate of change infixations. Fixation Rate (Frequency) (f_fixation) refers to the averagenumber of fixations per second (s{circumflex over ( )}(−1) or Hz)measured for all fixations, fixations over a period of time (e.g.f_(fixation|target present)) and/or fixations within a particular region(e.g. f_(fixation|display center)).

Fixations Count (Number) (N_(fixation)) refers to the number offixations measured for all fixations, fixations over a period of time(e.g. N_(fixation|target present)) and/or fixations within a particularregion (e.g. N_(fixation|display center)).

Saccade Position ([x₁,y₁, z₁|x₂, y₂, z₂]_(saccade) or [r₁, θ₁, φ₁|r₂,θ₂, φ₂]_(saccade)) refers to the starting (1) and ending (2) positionsof a saccadic eye movement in Cartesian or spherical 3D space. Thereference frame will generally be the same, within a given scenario, asthat used for gaze position. The saccade position definition includes,and is dependent upon, a rate of change, an initial saccade position,and a final saccade position.

Saccade Angle (Θ_(saccade)) refers to an angle describing the2-dimensional (ignoring depth) direction of a saccade with respect tosome reference in degrees (°) or radians (rad). Unless otherwisespecified the reference is vertically up and the angle increasesclockwise. The reference may be specified (e.g. (Θ_(saccade-target)) todenote the deviation of the saccade direction from some desireddirection (i.e. towards a target). The average saccade direction isdenoted with a bar Θ _(saccade) and may represent all or a subset ofsaccades (e.g. _Θ_(saccade|target present)); because the direction isangular (i.e. circular) the average direction may be random unless arelevant reference is specified (e.g. _Θ_(saccade−target|targetpresent)). The saccade angle may be used to determine how relevant atarget is to a user, also referred to as a context of relevancy towardsa target.

Saccade Magnitude (M_(saccade)) refers to the magnitude of a saccaderelating to the distance traveled; this may be given as a visual anglein degrees (°) or radians (rad), a physical distance with regard to theestimated gaze position (e.g. in centimeters (cm) or inches (in)) or adistance in display space with regard to the estimated gaze position ona display (e.g. in pixels (px)). In reference to a particular point (P)in space, the component of the saccade magnitude parallel to a directline to that point may be given as:M _(saccade-P) =M _(saccade)·cos(Θ_(saccade-P))

where M_(saccade) is the magnitude of the saccade and Θ_(saccade-P) isthe angle between the saccade direction and a vector towards point P.The average saccade magnitude is denoted with a bar M _(saccade), andthis notation may be applied to all saccades and/or a subset in time orspace and with regard to saccade magnitudes or the components of saccademagnitude relative to a designated point.

Pro-Saccade refers to movement towards some point in space, often atarget, area of interest or some attention-capturing event. By the aboveterminology a pro-saccade would have a relatively small saccadic angleand positive magnitude component relative to a designated position.

Anti-Saccade refers to movement away from some point in space, often dueto aversion or based on a task (instruction to look away). By the aboveterminology an anti-saccade would have a relatively large saccadic angle(around ±180° or ±π rad) and a negative magnitude component relative toa designated position.

Inhibition of Return (IOR) is related to anti-saccades and describes atendency during search or free viewing to avoid recently fixated regionswhich are less informative. IOR reflects a general strategy forefficient sampling of a scene. It may be furthered defined by, or afunction of, anti-saccades.

Saccade Velocity (v_(saccade)) or the velocity of a saccade is taken asthe change in magnitude over time (and not generally from magnitudecomponents towards a reference point). Based on the degree of magnitudeand direction of the saccade velocity, it may be indicative of a degreeof relevancy of the target to the user. The average saccade velocity isdenoted with a bar v_(saccade) and may be applied to all saccades or asubset in time and/or space.

Saccade Rate (Frequency) (v_(saccade)) denotes the average number ofsaccades per second (s⁻¹ or Hz) measured for all saccades, saccades overa period of time (e.g. f_(saccade|target present)), saccades within aparticular region (e.g. f_(saccade|display center)) and/or saccadesdefined by their direction (e.g. f_(saccade|towards target)).

Saccade Count (Number) (N_(saccade)) is the number of saccades measuredfor all saccades, saccades over a period of time (e.g. N_(saccade|targetpresent)), saccades within a particular region (e.g. N_(saccade|displaycenter)) and/or saccades defined by their direction (e.g.N_(saccade|towards target)).

Pursuit Eye Movements (PEM) is used to refer to both smooth pursuit eyemovements where gaze tracks a moving object through space andvestibulo-ocular movements that compensate for head or body movement. Itmay be further defined by data indicative of an initiation, a duration,and/or a direction of smooth PEM. Also included are compensatorytracking of stationary objects from a moving frame of reference. PEMgenerally do not consist of fixations and saccades but rathercontinuous, relatively slow motion interrupted by occasionalerror-correcting saccades. The smooth and saccadic portions of a PEMtrace may be subtracted and analyzed separately.

Body Tracking Definitions

Body tracking entails measuring and estimating the position of the bodyand limbs as a function of time and/or discrete events in timeassociated with a class of movement (e.g. a nod of the head).Information sources include video tracking with and without worn markersto aid in image processing and analysis, position trackers,accelerometers and various hand-held or worn devices, platforms, chairs,or beds.

Screen Distance (d_(screen)) refers to the distance between the user'seyes (face) and a given display device. As a static quantity, it isimportant for determining the direction towards various elements on thescreen (visual angle), but as a variable with time, screen distance canmeasure user movements towards and away from the screen. Screen distanceis dependent upon a rate of change, an initial position, and a finalposition between the user's eyes (face) and a given display device.Combined with face detection algorithms, this measure may be made fromdevice cameras and separate cameras with known position relative todisplays.

Head Direction (Facing) ([θ, ϕ]_(facing)) refers to the direction in 3Dpolar coordinates of head facing direction relative to either the bodyor to a display or other object in the environment. Tracked over timethis can be used to derive events like nodding (both with engagement andfatigue), shaking, bobbing, or any other form of orientation. Headdirection is dependent upon a rate of change, an initial position, and afinal position of head facing direction relative to either the body orto a display or other object in the environment.

Head Fixation, while similar to fixations and the various measuresassociated with eye movements, may be measured and behavior-inferred.Generally head fixations will be much longer than eye fixations. Headmovements do not necessarily indicate a change in eye gaze directionwhen combined with vestibulo-ocular compensation. Head fixation isdependent upon a rate of change, an initial position, and a finalposition of head fixations.

Head Saccade, while similar to saccades and their various measuresassociated with eye movements, may be measured as rapid, discrete headmovements. These will likely accompany saccadic eye movements whenshifting gaze across large visual angles. Orienting head saccades mayalso be part of auditory processing and occur in response to novel orunexpected sounds in the environment.

Head Pursuit, while similar to pursuit eye movements, tend to be slowerand sustained motion often in tracking a moving object and/orcompensating for a moving frame of reference.

Limb Tracking refers to the various measures that may be made of limbposition over time using video with image processing or worn/helddevices that are themselves tracked by video, accelerometers ortriangulation. This includes pointing devices like a computer mouse andhand-held motion controllers. Relative limb position may be used toderive secondary measures like pointing direction. Limb tracking isdependent upon a rate of change, an initial position, and a finalposition of the limbs.

Weight Distribution refers to the distribution of weight over a spatialarrangement of sensors while users stand, sit or lie down can be used tomeasure body movement, position and posture. Weight distribution isdependent upon a rate of change, an initial position, and a finalposition of weight.

Facial expressions including micro-expressions, positions of eyebrows,the edges, corners, and boundaries of a person's mouth, and thepositions of a user's cheekbones, may also be recorded.

Electrophysiological and Autonomic Definitions

Electrophysiological measures are based on recording of electricpotentials (voltage) or electric potential differences typically byconductive electrodes placed on the skin. Depending on the part of thebody where electrodes are placed various physiological and/or behavioralmeasures may be made based on a set of metrics and analyses. Typicallyvoltages (very small—microvolts μV) are recorded as a function of timewith a sample rate in the thousands of times per second (kHz). Whileelectrophysiological recording can measure autonomic function, othermethods can also be used involving various sensors. Pressuretransducers, optical sensors (e.g. pulse oxygenation), accelerometers,etc. can provide continuous or event-related data.

Frequency Domain (Fourier) Analysis allows for the conversion of voltagepotentials as a function of time (time domain) into waveform energy as afunction of frequency. This can be done over a moving window of time tocreate a spectrogram. The total energy of a particular frequency orrange of frequencies as a function of time can be used to measureresponses and changes in states.

Electroencephalography (EEG) refers to electrophysiological recording ofbrain function. Time averaged and frequency domain analyses (detailedbelow) provide measures of states. Combined with precise timinginformation about stimuli, event-related potentials (EEG-ERP) can beanalyzed as waveforms characteristic of a particular aspect ofinformation processing.

Frequency Bands are typically associated with brain activity (EEG) andin the context of frequency domain analysis different ranges offrequencies are commonly used to look for activity characteristic ofspecific neural processes or common states. Frequency ranges arespecified in cycles per second (s⁻¹ or Hz):

-   -   Delta—Frequencies less than 4 Hz. Typically associated with        slow-wave sleep.    -   Theta—Frequencies between 4 and 7 Hz. Typically associated with        drowsiness.    -   Alpha—Frequencies between 8 and 15 Hz.    -   Beta—Frequencies between 16 and 31 Hz.    -   Gamma—Frequencies greater than 32 Hz.

Electrocardiography (ECG) refers to electrophysiological recording ofheart function. The primary measure of interest in this context is heartrate.

Electromyography (EMG) refers to electrophysiological recording ofmuscle tension and movement. Measures of subtle muscle activation, notnecessarily leading to overt motion, may be made. Electrodes on the facecan be used to detect facial expressions and reactions.

Electrooculography (EOG) refers to electrophysiological recording acrossthe eye. This can provide sensitive measures of eye and eyelid movement,however with limited use in deriving pupil position and gaze direction.

Electroretinography (ERG) refers to electrophysiological recording ofretinal activity.

Galvanic Skin Response (GSR) (Electrodermal response) is a measure ofskin conductivity. This is an indirect measure of the sympatheticnervous system as it relates to the release of sweat.

Body Temperature measures may be taken in a discrete or continuousmanner. Relatively rapid shifts in body temperature may be measures ofresponse to stimuli. Shifts may be measured by tracking a rate of changeof temperature, an initial temperature, and a final temperature.

Respiration Rate refers to the rate of breathing and may be measuredfrom a number of sources including optical/video, pneumography andauditory and will typically be measured in breaths per minute (min⁻¹Brief pauses in respiration (i.e. held breath) may be measured in termsof time of onset and duration.

Oxygen Saturation (S_(O) ₂ ) is a measure of blood oxygenation and maybe used as an indication of autonomic function and physiological state.

Heart Rate is measured in beats per minute (min⁻¹ nd may be measuredfrom a number of sources and used as an indication of autonomic functionand physiological state.

Blood Pressure is typically measured with two values: the maximum(systolic) and minimum (diastolic) pressure in millimeters of mercury(mm Hg). Blood pressure may be used as an indication of autonomicfunction and physiological state.

Efferent Audio Recording Definitions

Audio recording from nearby microphones can measure behavioral and evenautonomic responses from users. Vocal responses can provide measures ofresponse time, response meaning or content (i.e. what was said) as wellas duration of response (e.g. “yeah” vs. “yeeeeeeeaaaah”). Otherutterances like yawns, grunts or snoring might be measured. Otheraudible behaviors like tapping, rocking, scratching or generally fidgetybehavior may be measured. In certain contexts, autonomic behaviors likerespiration may be recorded.

Vocalizations, such as spoken words, phrases and longer constructionsmay be recorded and converted to text strings algorithmically to derivespecific responses. Time of onset and duration of each component(response, word, syllable) may be measured. Other non-lingual responses(yelling, grunting, humming, etc.) may also be characterized.Vocalizations may reflect a range of vocal parameters including pitch,loudness, and semantics.

Inferred Efferent Responses refer to certain efferent responses ofinterest that may be recorded by audio and indicate either discreteresponses to stimuli or signal general states or moods. Behaviors ofinterest include tapping, scratching, repeated mechanical interaction(e.g. pen clicking) bouncing or shaking of limbs, rocking and otherrepetitive or otherwise notable behaviors.

Respiration, such as measures of respiration rate, intensity (volume)and potentially modality (mouth vs. nose) may also be made.

Afferent Classification/Definitions

The states discussed below are generally measured in the context of orresponse to various stimuli and combinations of stimuli andenvironmental states. A stimulus can be defined by the afferent inputmodality (visual, auditory, haptic, etc.) and described by its features.Features may be set by applications (e.g. setting the position, size,transparency of a sprite displayed on the screen) or inferred byimage/audio processing analysis (e.g. Fourier transforms, saliencymapping, object classification, etc.).

Regions of interest as discussed below may be known ahead of time andset by an application, may be defined by the position and extent ofvarious visual stimuli and/or may be later derived after data collectionby image processing analysis identifying contiguous, relevant and/orsalient areas. In addition to stimulus features, efferent measures maybe used to identify regions of interest (e.g. an area where a user tendsto fixate is defined by gaze position data). Likewise both afferent andefferent measures may be used to segment time into periods for summaryanalysis (e.g. total number of fixations while breath is held).

Sensory Data Exchange Platform Overview

Reference is made to FIG. 1 , which shows a block diagram 100illustrating user interaction with an exemplary SDEP, in accordance withan embodiment of the present specification. In an embodiment, a user 102interfaces with a media system, such as an app on a tablet computer or aVR/AR/M×R system 104. The media system 104 may include devices such asHMDs, sensors, and/or any other forms of hardware elements 106 thatpresent visual, auditory, and other sensory media to the user andenables collection of user response data during user interaction withthe presented media. The media may be communicated by a server, througha network, or any other type of content platform that is capable ofproviding content to hardware devices, such as HMDs. Sensors may bephysiological sensors, biometric sensors, or other basic and advancedsensors to monitor user 102. Additionally, sensors may includeenvironmental sensors that record audio, visual, haptic, or any othertypes of environmental conditions that may directly or indirectly impactthe vision performance of user 102. The media system 104 may alsoinclude software elements 108 that may be executed in association withhardware elements 106. Exemplary software elements 108 include gamingprograms, software applications (apps), or any other types of softwareelements that may contribute to presentation of media to user 102.Software elements 108 may also enable the system to collect userresponse data. Collected data may be tagged with information about theuser, the software application, the game (if any), the media presentedto the user, the session during which the user interacted with thesystem, or any other data. A combination of hardware elements 106 andsoftware elements 108 may be used to present media to user 102.

In an embodiment, stimulus and response data collected from user's 102interaction with the software system 104 may constitute data sources110. Data sources 110 may be created within a SDEP 118 based on aninteraction between software elements 108 and SDEP 118. Softwareelements 108 may also interact with SDEP 118 through proprietaryfunction calls included in a Software Development Kit (SDK) fordevelopers (i.e. the developers may send/receive data to/from SDEP 118using predefined functions). SDEP 118 may include storage and processingcomponents and could be a computing system. The functionality of SDEP118 may largely reside on one or more servers and the data stored andretrieved from cloud services. Sources of data may be in the form ofvisual data, audio data, data collected by sensors deployed with thesoftware system 104, user profile data, or any other data that may berelated to user 102. Visual data may largely include stimulus data andmay be sourced from cameras (such as cell phone cameras or other visionequipment/devices), or from other indirect sources such as games andapplications (apps). Sensors may provide spatial and time series data.User data may pertain to login information, or other user-specificinformation derived from their profiles, from social media apps, orother personalized sources. In embodiments, data sources are broadlyclassified as afferent data sources and efferent data sources, which aredescribed in more detail in subsequent sections of the specification. Inan embodiment, user profile data may be collected from another database,or may be provided through a different source. In an exemplaryembodiment user profile data may be provided by service providersincluding one or more vision care insurance provider. In otherembodiments, the user profile data may be collected from other sourcesincluding user's device, opt-in options in apps/games, or any othersource.

Data sources 110 may be provided to a data ingestion system 112. Dataingestion system 112 may extract and/or transform data in preparation toprocess it further in a data processing system 114. Data adapters, whichare a set of objects used to communicate between a data source and adataset, may constitute data ingestion system 112. For example, an imagedata adapter module may extract metadata from images, and may alsoprocess image data. In another example, a video data adapter module mayalso extract metadata from video data sources, and may also include avideo transcoder to store large volumes of video into distributed filesystem. In another example, a time series data adapter module parsessensor data to time series. In another embodiment, a spatial dataadapter module may utilize data from relatively small areas such asskin, and spatially transform the data for area measurements. In anotherexample, a user profile data adapter module may sort general user data,such as through a login, a social media connect API, unique identifierson phone, and the like.

SDEP 118 may further comprise a data processing system 114 that receivesconditioned data from data ingestion system 112. A machine learningmodule 152 within data processing system 114 may communicate with astorage 154 and a real time queue 156 to output data to a data servingsystem 116, which may include an Application Program Interface (API). Inembodiments, the machine learning system may implement one or more knownand custom models to process data output from data ingestion system 112.

In embodiments, SDEP 118 may further include a module 120 for backendanalytics that feeds another API 122. API 122 may, in turn, interfacewith user 102, providing modified media to user 102.

FIG. 2A is a block diagram illustrating processing of a sensor datastream before it reaches a query processor, in accordance with anembodiment of the present specification. In an embodiment, FIG. 2Aillustrates a lambda architecture 200 for a sensor data stream receivedby a SDEP. Data processing architecture 200 may be designed to handlelarge quantities of data by parallel processing of data stream andbatch. In an embodiment, a sensor data stream 202 comprising sensor datacollected from users in real time is provided to a real time layer 204.Real time layer 204 may receive and process online data through a realtime processor 214. Data collected in batches may be provided to a batchlayer 206. Batch layer 206 comprises a master data set 222 to receiveand utilize for processing time stamped events that are appended toexisting events. Batch layer 206 may precompute results using adistributed processing system involving a batch processor 216 that canhandle very large quantities of data. Batch layer 206 may be aimed atproviding accurate data by being able to process all available sensordata, to generate batch views 218. A bulk uploader 220 may upload outputto be stored in a database 210, with updates completely replacingexisting precomputed batch views. Processed data from both layers may beuploaded to respective databases 208 and 210 for real time serving andbatch serving. Data from databases 208 and 210 may subsequently beaccessed through a query processor 212, which may be a part of a servinglayer. Query processor 212 may respond to ad-hoc queries by returningprecomputed views or building views from the processed data. Inembodiments, real-time layer 204, batch layer 206, and serving layer maybe utilized independently.

Data Acquisition

Events may be coded within the stream of data, coming potentially fromthe app, the user and environmental sensors, and may bear timestampsindicating when things happen. Anything with an unambiguous time ofoccurrence may qualify as an “event”. Most events of interest may bediscrete in time, with time stamps indicating either the start or theend of some state. As an exception, electrophysiological data may berecorded continuously and generally analyzed by averaging segments ofdata synchronized in time with other events or by some other analysis.

In an embodiment, data collected from interactions with user 102 isbroadly classified as afferent data and efferent data, corresponding toafferent events and efferent events. In the peripheral nervous system,an afferent nerve fiber is the nerve fiber (axon) of an afferent neuron(sensory neuron). It is a long process (projection) extending far fromthe nerve cell body that carries nerve impulses from sensory receptorsor sense organs toward the central nervous system. The oppositedirection of neural activity is efferent conduction. Conversely, anefferent nerve fiber is the nerve fiber (axon) of an efferent neuron(motor neuron). It is a long process (projection) extending far from thenerve cell body that carries nerve impulses away from the centralnervous system toward the peripheral effector organs (mainly muscles andglands).

A “stimulus” may be classified as one or more events, typicallyafferent, forming a discrete occurrence in the physical world to which auser may respond. A stimulus event may or may not elicit a response fromthe user and in fact may not even be consciously perceived or sensed atall; thus, if an event occurred, it is made available for analysis.Stimulus event classes may include “Application Specific Events” and“General and/or Derived Stimulus Events”.

Application Specific Events may include the many stimulus event classesthat may be specific to the sights, sounds, and other sensory effects ofa particular application. All of the art assets are potential visualstimuli, and all of the sound assets are potential auditory stimuli.There may be other forms of input including, but not limited togustatory, olfactory, tactile, along with physiologic inputs—heart rate,pulse ox, basal body temperature, along with positionaldata—accelerometer, visual-motor—limb movement, gyroscope—headmovements/body movement—direction, force, and timing. The sudden orgradual appearance or disappearance, motion onset or offset, playing orpausing or other change in state of these elements will determine theirspecific timestamp. Defining these stimulus event classes may require anapp developer to collaborate with the SDE, and may include specificdevelopment of image/audio processing and analysis code.

General and/or Derived Stimulus Events are those stimulus events thatmay be generic across all applications. These may include those afferentevents derived from video (e.g. head mounted camera) or audio datarecorded of the scene and not coming directly from the app (which itselfwill provide a more accurate record of those events). Device specific,but not app specific, events may also be classified. Likewisecalibration and other activities performed for all apps may beconsidered general (though perhaps still able to be categorized by theapp about to be used).

Some stimulus events may not be apparent until after a large volume ofdata is collected and analyzed. Trends may be detected and investigatedwhere new stimulus event classes are created to explain patterns ofresponding among users. Additionally, descriptive and predictiveanalysis may be performed in order to facilitate real-time exchange ofstimuli/content depending on the trends/patterns so as to personalizeuser-experience.

A “response” may be classified as one or more events, typicallyefferent, forming a discrete action or pattern of actions by the user,potentially in response to a perceived stimulus (real or imagined).Responses may further include any changes in physiological state asmeasured by electrophysiological and/or autonomic monitoring sensors.Responses may not necessarily be conscious or voluntary, though theywill be identified as conscious/unconscious and voluntary/involuntarywhenever possible. Response events classes may include discreteresponses, time-locked mean responses, time derivative responses, and/orderived response events.

“Discrete Responses” represent the most common response eventsassociated with volitional user behavior and are discrete in time with aclear beginning and end (usually lasting on the order of seconds ormilliseconds). These include, among others, mouse or touch screeninputs, vocalizations, saccadic and pursuit eye movements, eye blinks(voluntary or not), head or other body part movement andelectrophysiologically detected muscle movements.

Due to the noisy nature of some data recording, notablyelectrophysiological recording, it is difficult to examine responses toindividual stimulus events. A Time-Locked Mean Response refers to thepattern of responding to a particular stimulus event, which may beextracted from numerous stimulus response events by averaging. Data fora length of time (usually on the order of seconds) immediately followingeach presentation of a particular stimulus is put aside and thenaveraged over many “trials” so that the noise in the data (presumablyrandom in nature) cancels itself out leaving a mean response whosecharacteristics may be measured.

Time Derivative Responses reflect that some responses, particularlyautonomic responses, change slowly over time; Sometimes too slowly toassociate with discrete stimulus events. However the average value,velocity of change or acceleration of velocity (and other derivedmeasures) within certain periods of time may be correlated with othermeasured states (afferent or efferent).

As with stimulus events, some response events may not be apparent beforedata collection but instead reveal themselves over time. Whether throughhuman or machine guided analysis, some characteristic responses mayemerge in the data, hence may be termed Inferred Response Events.

Whenever possible, responses will be paired with the stimuli which (mayhave) elicited them. Some applications may make explicit in the datastream how stimuli and responses are paired (as would be the case inpsychophysical experimentation). For the general case, stimulus eventclasses will be given a set period of time, immediately followingpresentation, during which a response is reasonably likely to be made.Any responses that occur in this time frame may be paired with thestimulus. If no responses occur then it will be assumed the user did notrespond to that stimulus event. Likewise response events will be given aset period of time, immediately preceding the action, during which astimulus is likely to have caused it. Windows of time both after stimuliand before responses may be examined in order to aid in the discovery ofnew stimulus and response event classes not previously envisioned.

Stimulus and Response Event Classes may be defined and differentiated bytheir features (parameters, values, categories, etc.). Some features ofan event class may be used to establish groups or categories within thedata. Some features may (also) be used to calculate various metrics.Features may be numeric in nature, holding a specific value unique tothe event class or the individual instance of an event. Features may becategorical, holding a named identity either for grouping or potentiallybeing converted later into a numerical representation, depending on theanalysis.

The features of stimulus events may primarily constitute a physicaldescription of the stimulus. Some of these features may define the eventclass of the stimulus, and others may describe a specific occurrence ofa stimulus (e.g. the timestamp). The named identity of a stimulus (e.g.sprite file name) and state information (e.g. orientation or pose) arestimulus features. The pixel composition of an image or waveform of asound can be used to generate myriad different descriptive features of astimulus. Some stimulus features may require discovery through dataanalysis, just as some stimulus event classes themselves may emerge fromanalysis.

Response features may generally include the type or category of responsemade, positional information (e.g. where the mouse click occurred orwhere a saccade originated/landed, a touch, a gaze, a fixation, turn ofhead, turn of body, direction and velocity of head, or body/limbmovement) and timing information. Some derived features may come fromexamining the stimulus to which a response is made; for example: whetherthe response was “correct” or “incorrect”.

FIG. 2B illustrates an exemplary outline of a data analysis chain. Thedata analysis begins at the lowest level at 232 wherein data at thislevel may not be simplified or broken down further. At 232, parametersof a single stimulus can be used for multiple measures based ondifferent independent variables, which correspond to direct features ofa stimulus. Parameters of a single response can be used for multiplemeasures based on different dependent variables.

FIG. 3 illustrates an overview 300 of sources of digital data. Inembodiments, afferent data 304 may be collected from sources thatprovide visual information 307, auditory information 308, spatialinformation 310, or other environmentally measured states including andnot limited to temperature, pressure, and humidity. Sources of afferentdata 304 may include events that are meant to be perceived by a user302. User 302 may be a user interfacing with a media system inaccordance with various embodiments of the present specification.

Afferent and efferent data may be collected for a plurality of peopleand related to demographic data that correspond to the profiles for eachof the plurality of people, wherein the demographic data includes atleast the sex and the age of each of the plurality of people. Once sucha database is created, medical treatments can be created that aretargeted to a group of people having at least one particular demographicattribute by causing the media content of that service to have a greaterimpact on the retino-geniculo-cortical pathway of the targeted group.

Afferent Data

Afferent (stimulus) events may be anything happening on a displayprovided to user 302 in the display, events coming from speakers orhead/earphones, or haptic inputs generated by an app. Data may also becollected by environment sensors including and not limited tohead-mounted cameras and microphones, intended to keep a record ofthings that may have been seen, heard, or felt by user 302 but notgenerated by the app itself. Afferent data 304 may be a form ofstimulus, which may be broken down into raw components (features orfeature sets) that are used to build analytic metrics.

In embodiments, an afferent (stimulus) event is paired with an efferent(response) event. In the pairing, each of the component stimulusfeatures may be paired with each of the component response features foranalysis. In some cases pairs of stimulus features or pairs of responsefeatures may also be examined for correlations or dependencies.Stimulus/response feature pairs are at the root of most of theconceivable metrics to be generated. All analyses may be broken down bythese feature pairs before being grouped and filtered according tovarious other of the event features available. In embodiments, for alldata sources including afferent 304 and efferent 306 data sources,timing information is required to correlate inputs to, and outputs from,user's 302 sensory system. The correlations may be utilized to identifycharacteristic metrics or psychophysical metrics for the user. Forexample, if the media system 104 records that an object was drawn on ascreen at time tS (stimulus), and also that a user pressed a particularkey at a time tR (response), the time it took the user to respond to thestimulus may be derived by subtracting tR−tS. In alternate embodiments,the user may press a key, or make a gesture, or interact with the mediaenvironment through a touch or a gesture. This example correlatesafferent data 302 and efferent data 304.

An example that correlates two types of afferent data 304 may be if agaze tracker indicates that the gaze position of a user changed smoothlyover a given period of time indicating that the user was tracking amoving object. However, if a head tracker also indicates smooth motionin the opposite direction, at the same time, it might also indicate thatthe user was tracking a stationary object while moving their head.

Another example that correlates two types of afferent data 304 may be ifvisual object appears at time t1, and a sound file is played at time t2.If the difference between t1 and t2 is small (or none), they may beperceived as coming from the same source. If the difference is large,they may be attributed to different sources.

The data taken from accumulated response events may be used to describepatterns of behavior. Patterns of responding, independent of whatstimuli may have elicited them, can be used to categorize variousbehavioral or physiological states of the user. Grouping responses bythe stimuli that elicited them can provide measures of perceptualfunction. In some cases analyses of stimulus events may provide usefulinformation about the apps themselves, or in what experiences userschoose to engage. The analysis may include following parameters: uniqueevents, descriptive statistics, and/or psychometric functions.

Unique Events represent instances where raw data may be of interest.Some uncommon stimulus or response events may not provide opportunitiesfor averaging, but instead are of interest because of their rarity. Someevents may trigger the end of a session or time period of interest (e.g.the user fails a task and must start over) or signal the beginning ofsome phase of interaction.

Descriptive Statistics provide summarized metrics. Thus, if multipleoccurrences of an event or stimulus/response event or feature pairingmay be grouped by some commonality, measures of central tendency (e.g.mean) and variability (e.g. standard deviation) may be estimated. Thesesummarized metrics may enable a more nuanced and succinct description ofbehavior over raw data. Some minimal level of data accumulation may berequired to be reasonably accurate.

Psychometric Functions may form the basis of measures of perceptualsensitivity and ability. Whenever a particular class of stimulus eventis shown repeatedly with at least one feature varying amongpresentations there is an opportunity to map users' pattern of responsesagainst that stimulus feature (assuming responding varies as well). Forexample, if the size (stimulus feature) of a particular object in a gamevaries, and sometimes the user finds it and sometimes they don't(response feature), then the probability of the user finding that objectmay be plotted as a function of its size. This may be done for multiplestimulus/response feature pairs for a single stimulus/response eventpairing or for many different stimulus/response event pairs that happento have the same feature pairing (e.g. size/detection). When a responsefeature (detection, discrimination, preference, etc.) plotted against astimulus feature (size, contrast, duration, velocity, etc.) is availablewith mean responses for multiple stimulus levels, a function to thatdata (e.g. detection vs. size) may be fitted. The variables thatdescribe that function can themselves be descriptive of behavior.Thresholds may be defined where on one side is failure and the otherside success, or on one side choice A and the other side choice B, amongothers.

Visual Information

Referring back to FIG. 3 , in an embodiment, for an application, visualinformation data 307 from physical display(s) and the visual environmentis in the form of still image files and/or video files captured by oneor more cameras. In an embodiment, data is in the form of instructionsfor drawing a particular stimulus or scene (far less data volumerequired, some additional time in rendering required).

FIG. 4 is a block diagram 400 illustrating characteristic metrics forvisual data, in accordance with an embodiment of the presentspecification. Characteristic metrics may characterize a user sessionand may be time-averaged. Referring FIG. 4 , scope 402 may refer towhether the visual data is for an entire scene (the whole visual displayor the whole image from a user-head-mounted camera). Physical attributes404 may refer to objective measures of the scene or objects within it.They may include location relative to the retina, head and body, anorthogonal 3-D chromoluminance; and contrast vs. spatial frequency vs.orientation. Categorical attributes 406 may be named properties of theimage, which may include named identity of an object, and/or the groupidentity.

Visual stimuli may generally be taken in as digital, true color images(24-bit) either generated by an application (image data provided by appdirectly) or taken from recorded video (e.g. from a head mountedcamera). Images and video may be compressed in a lossy fashion; whereweighted averaging of data may account for lossy compression, butotherwise image processing would proceed the same regardless. Adeveloper may choose to provide information about the presentation of astimulus which may allow for the skipping of some image processing stepsand/or allow for post hoc rendering of scenes for analysis. Visualstimuli may include, but are not limited to the following components:objects, size, chromatic distance, luminance contrast, chromaticcontrast, spatial feature extraction, saliency maps and/or temporaldynamics.

Objects (stimuli) may be identified in an image (or video frame) eitherby information from the application itself or found via machine learning(Haar-like features classification cascade, or similar). Onceidentified, the pixels belonging to the object itself (or within abounding area corresponding to a known size centered on the object) willbe tagged as the “object”. The pixels in an annulus around the object(necessarily within the boundaries of the image/scene itself) with thesame width/height of the object (i.e. an area 3× the object width and 3×the object height, excluding the central area containing the object)will be tagged as the “surround”. If another image exists of the sameexact area of the surround, but without the object present (thus showingwhat is “behind” the object), that entire area without the object may betagged as the “background”. Metrics may be calculated relative to thesurround and also relative to the background when possible. Objectsegments or parts may be used to break objects down into other objectsand may also be used for identity or category variables. Objects neednot correspond to physical objects and may include regions or boundarieswithin a scene or comprise a single image feature (e.g. an edge).

Object size is an important feature for determining acuity, or fromknown acuity predicting whether a user will detect or correctly identifyan object. The object size may be defined as a width and height, eitherbased on the longest horizontal and vertical distance between pixellocations in the object or as the width and height of a rectangularbounding box defining the object's location. Smaller features that maybe necessary to successfully detect or discriminate the object fromothers may be located within the object. It may be assumed that thesmallest feature in an object is 10% of the smaller of its twodimensions (width and height). It may also be assumed the smallestfeature size is proportional to the size of a pixel on the display for agiven viewing distance. The smallest feature size may be more explicitlyfound either by analysis of a Fourier transform of the image orexamining key features from a Harr-like feature classification cascade(or similar machine learning based object detection) trained on theobject.

The first of two breakdowns by color, chromatic distance is a measure ofthe color difference between the object and its surround/background,independent of any luminance differences. Red, green and blue values maybe independently averaged across all pixels of the object and all pixelsof the surround/background. These mean RGB values will be converted intoCIE Tristimulus values (X, Y and Z) and then into CIE chromaticity (xand y) using either standard conversion constants or constants specificto the display used (when available). In an embodiment, conversionconstants for conversion from RGB to XYZ, taken from Open CV function‘cvtColor’ based on standard primary chromaticities, a white point atD65, and a maximum, white luminance of 1, is:

$\left. \begin{bmatrix}X \\Y \\Z\end{bmatrix}\leftarrow{\begin{bmatrix}0.412453 & 0.357580 & 0.180423 \\0.212671 & 0.715160 & 0.072169 \\0.019334 & 0.119193 & 0.950227\end{bmatrix} \cdot \begin{bmatrix}R \\G \\B\end{bmatrix}} \right.$

In this embodiment, RGB is converted to xy using the following:

$x = \frac{X}{X + Y + Z}$ $y = \frac{Y}{X + Y + Z}$

The absolute distance between the chromaticity of the object and that ofthe surround/background will be logged as the chromatic distance. Next,a line will be drawn from the midpoint between the two chromaticitiesand each of the three copunctal points for L, M and S cones. These linesare confusion lines for L, M and S cone deficiencies, along whichsomeone missing one of those cone types would be unable to discriminatechromaticity. The component of the line between object andsurround/background chromaticity parallel to each of these threeconfusion lines will be logged as the L, M and S specific chromaticdistances.

FIG. 5 provides a graphical presentation of color pair confusioncomponents, in accordance with an embodiment of the presentspecification. Referring to the figure, a line 508 is drawn between thetwo chromaticities given. As seen in the figure, three large dots—red502, green 504, and blue 506 are copunctal points for L, M and S cones,respectively. From each dot extends a similarly color-coded, dashedline. Bold line 508 has a mid-point where the three, dashed linesintersect. Based on the angle between line 508 and the lines drawn fromthe midpoint to each of the copunctal points, the parallel component ofthat line for each of the three resulting confusion lines is determined.In embodiments, the closer to the parallel line between the colors is toa particular confusion line, the more difficult it will be for someonewith a deficiency of the corresponding cone to discriminate. Thecomponent length divided by the total length (the quotient will be inthe interval [0,1]) would be roughly the probability of the colors beingconfused.

FIG. 6 shows a graph illustrating how luminance may be found for a givenchromaticity that falls on the top surface of the display gamutprojected into 3D chromoluminance space. The graph shows a projection ofa full display gamut for a computer screen into CIE 1931 chromoluminancespace. While the RGB space used to define the color of pixels on adisplay can be represented by a perfect cube, the actual physicalproperty of luminance is somewhat complexly derived from those values,represented by the shape seen in FIG. 6 . Luminance contrast may bedefined in three ways. Generally the context of an analysis will suggestwhich one of the three to use, but all three may be computed for anyobject and its surround/background. For instances where a small objectis present on a large, uniform background (e.g. for text stimuli), Webercontrast may be computed using the CIE Tristimulus values Y(corresponding to luminance) calculated from the mean RGB of the objectand of the surround/background. Here it is assumed that the averageluminance is roughly equal to the surround luminance. Weber contrast canbe positive or negative and is theoretically unbounded. Forobject/surrounds that are periodic in nature, and especially withgradients (e.g. a sine wave grating), Michelson contrast may be computedfrom the minimum and maximum luminance values in the stimulus. Michelsoncontrast will always be a value between 0 and 1. For most cases it willbe necessary to compute contrast from all of the pixel values, insteadof from a mean or from the minimum and maximum. The RMS contrast (rootmean square, or standard deviation) can be found by taking the standarddeviation of the CIE Tristimulus value Y for all pixels. The RMScontrast of the object is one measure. The RMS contrast of the objectrelative to the RMS contrast of the surround/background is another.Finally, the RMS contrast of the object and surround together is yet athird measure of RMS contrast that can be used.

Chromatic contrast may be calculated on any pair of chromaticity values,independently, in all of the ways described above for luminancecontrast. The most useful of these will either be the a* and b*components of CIELAB color space, or the L vs. M and S vs. LM componentsof cone-opponent color space. For any pair of dimensions, the Weber,Michelson and/or RMS contrast may be calculated, depending on the typeof stimulus being analyzed. In addition, RMS contrast will be calculatedfor L, M and S cone deficiencies. CIE chromaticity values for all pixelswill be converted into three sets of polar coordinates centered on theL, M and S copunctal points. In an embodiment, the following equation isused to convert Cartesian coordinates to polar coordinates, with anoption to provide center points other than [0,0]:

$\theta = {\tan^{- 1}\left( \frac{y - y_{c}}{x - x_{c}} \right)}$${Radius} = \sqrt{\left( {y - y_{c}} \right)^{2} + \left( {x - x_{c}} \right)^{2}}$

RMS contrast may be calculated based on the radius coordinates for eachconversion.

In addition to finding objects, algorithms may also identify prominentfeatures present in a scene, or within objects, that may captureattention, be useful for a task the user is performing or otherwise beof interest as independent variables to correlate with behavior. Edges,those inside identified objects and otherwise, may be targets forfixations or other responses and their positions may be responsible forobserved positional errors in responding and be worth correlating withcorrect and incorrect responses. Regions, contours, surfaces,reflections, shadows and many other features may be extracted from thisdata.

Saliency Maps refer to data that are collected from user interactions toinform models of saliency for future analysis of stimulus scenes. Edges,contours and other image features may be used to measure saliency andpredict where user responses, including eye gaze fixations, may fall.Multiple algorithms may be applied to highlight different types offeatures in a scene.

Temporal Dynamics are also important because features of a visualdisplay or environment, and any objects and object features thereof, maychange over time. It will be important to log the time of any change,notably: appearance/disappearance or change in brightness/contrast ofobjects or features, motion start/stop or abrupt position change (in x,y, z planes), velocity change (or acceleration or any higher order timederivative of position) and any and all changes in state or identity ofobjects or features. Changes in chromaticity or luminance of objects orfeatures should also be logged. Secondary changes in appearanceresulting from changes in orientation or pose of an object or theobject's position relative to the surround/background may also belogged.

Auditory Information

Referring back to FIG. 3 , auditory information 308 may be received fromaudio output such as speakers, and the environment by using microphones.In an embodiment auditory information 308 may be available in raw,waveform files or in more descriptive terms (e.g. this audio file playedat this time).

FIG. 7 illustrates characteristic metrics 700 for auditory information308, in accordance with an embodiment of the present specification.Referring to FIG. 7 , a positional reference 702 may be noted toidentify the location of sounds. The position, relative to a user'shead, of an object or speaker in the environment will vary as they movetheir head. The position of a virtual source perceived throughheadphones may not change as the user turns their head (unless headtracking and sound processing work together to mimic those changes).

The physical attributes 704 of sound may include their location (derivedfrom intensity, timing and frequency differences between the ears),frequency composition (derived from the waveform), and the compositionof different sources. Categorical attributes 706 may be named propertiesof the image, which may include named identity of an object, and/or thegroup identity and may follow a similar description as for visualstimuli.

Auditory (Sound) stimuli may generally be taken in as digital waveforms(with varying spatial and temporal resolution or bitrate and possiblecompression) either generated by an application or taken from recordedaudio (e.g. head mounted microphones, preferably binaural). Compressionparameters, if any, may be recorded. Developers may choose to provideinformation about the presentation of a stimulus which may allow for theskipping of some processing. Visual information may be used to model theaudio environment so that sound reflections or obscurations can be takeninto account. Audio stimuli may be broken down to include the followingparameters: Fourier Decomposition, Head-Centric Position, SoundEnvironment, and/or Objects.

Fourier Decomposition may be performed to break sound waves intocomponents based on sound objects. Time-domain waveform data may betransformed into the frequency domain such that the amplitude and phaseof different audio frequencies over time may be analyzed. This willallow the utilization of sound parameters (e.g. frequency, amplitude,wavelength, shape and envelope, timbre, phase, etc.) as independentvariables.

Head-Centric Position or head tracking data may be necessary forenvironmental sounds. The position of sound sources relative to a user'sears may be derived, and whenever possible the sound waveforms as theyexist at the user's ears may be recorded (ideally from binaural,head-mounted microphones). Binaural headset sound sources (e.g.headphones/earphones) may obviate the necessity for this.

Similarly, tracking data for body and/or limbs may be necessary forenvironmental sounds. The position of sound sources relative to a user'sbody and limbs may be derived. This data may be related to head trackingdata identified for environmental sounds. The data may enableunderstanding of how body and limbs react with the movement of head.

Sound Environment is not critical in most common use cases (e.g. soundis coming from headset or from directly in front of the user), but willbe important for considering environmental sounds to which users areanticipated to respond. Objects in the environment that reflect and/orblock sound (commonly frequency specific) may change the apparent sourcelocation and other frequency dependent features of a sound. It may beuseful to roughly characterize the physical environment as it affectsthe propagation of sound from its sources to the user.

Audio objects may be detected and segmented out using the same type ofmachine learning algorithms (Haar-like feature classification cascadesor similar) that are used for detecting and segmenting out visualobjects. This should be used whenever possible to obtain accurate audioevent details and may also be useful for extracting audio parametersused by the auditory system for localization.

Most analysis may revolve around visual and (to a lesser extent)auditory stimuli occurring discretely in time. Other stimuli may includethose sensed in other modalities (e.g. touch, taste, smell, etc.) orgeneral environmental state variables that define the context of userinteractions with applications (e.g. ambient lighting and backgroundaudio).

Examples of other stimuli may include the following:

Haptic Stimuli, where developers may choose to use haptic feedbackmechanisms and, if they so choose, provide details about the nature andtiming of those events. Haptic stimulation may also be derived viadirect recording (unlikely) or derived from other sources (e.g. hearingthe buzz of a physical vibration via microphone).

Other Modality Stimuli, where developers may be able to initiate smell,taste, temperature, pressure, pain or other sensation at discrete timescreating stimulus events not already discussed. As with haptic stimuli,any record of such stimulation would best come directly from theapplication itself via function calls.

Environmental Stimuli, or stimuli that do not occur discretely in timeand are either of constant state or steadily repeating, may provideimportant context for the discrete stimuli and responses that occur in asession. Ambient light levels may affect contrast sensitivity, baselinepupil size, circadian patterns and other physiological states of theuser. Ambient sounds may affect auditory sensitivity, may mask certainauditory stimuli and also affect physiological and other states of theuser. The time of day may also be an important variable forcategorization and correlation. Though perhaps not readily recorded byan application, user input could provide information about sleeppatterns, diet and other physiologically relevant state variables aswell as categorical descriptions of the space including temperature,pressure, humidity (which may also be derived from location and otherservices).

Spatial Information

Referring back to FIG. 3 , in an embodiment, spatial information 310 mayconsist of descriptions of the setting around user 302. This may includespatial orientation of user 302 and physical space around user 302.

In an embodiment, setting is an environment in which interactionsbetween user 302 and the app take place. Setting data may refer tothings that are mostly static during a session including the physicalsetting, ambient light levels, room temperature, and other types ofsetting information. In embodiments, spatial information 310 is a partof the setting data. Setting data may generally be constant throughout asession with user 302 and therefore may not be broken down into “events”as described earlier. Setting data may pertain to a physical setting ormay relate to personal details of user 302.

Physical setting data may correspond to any description of the physicalspace, such as and not limited to a room or an outdoor setting, and maybe useful to categorize or filter data. In an exemplary embodiment,physical setting data such as the ambient lighting present, may directlyaffect measures of pupil size, contrast sensitivity and others. Lightingmay affect quality of video eye tracking, as well as any afferent eventsderived from video recording of a scene. Similarly, environmental soundsmay affect users' sensitivity as well as the ability to characterizeafferent events derived from audio recording.

Personal details of a user may pertain to any personal, largelydemographic, data about the user or information about their presentphysiological or perceptual state (those that will remain largelyunchanged throughout the session). This data may also be useful forcategorization and filtering. Personal details may include anyinformation regarding optics of the user's eyes (for example, thosederived from knowledge of the user's eyeglass or contact prescription).Personal details may also include diet related information, such asrecent meal history. Further, time, duration, and quality of most recentsleep period, any psychoactive substances recently taken in (e.g.caffeine) and recent exercise or other physical activity may all impactoverall data.

Efferent Data

Eye Tracking

Video eye tracking and electrooculography provide information about eyemovements, gaze direction, blinking and pupil size. Derived from theseare measures of vergence, fatigue, arousal, aversion and informationabout visual search behavior. Information pertaining to eye movementsinclude initiation, duration, and types of pro-saccadic movements(toward targets), anti-saccadic movements (toward un-intended target),the amount of anti-saccadic error (time and direction from intended tounintended target), smooth pursuit, gaze with fixation duration, pupilchanges during movement and during fixation, frequency and velocity ofblink rate, as well as frequency and velocity of eye movements.Information pertaining to vergence may include both convergence anddivergence—in terms of initiation and duration. Combined withinformation about the visual scene, measures of accuracy, search timeand efficiency (e.g. minimizing number of saccades in search) can bemade.

Autonomic measures derived from video eye tracking data may be used toguide stimulus selection towards those that increase or decrease arousaland/or aversion. Summary information about gaze position may indicateinterest or engagement and likewise be used to guide stimulus selection.

Referring to FIG. 3 , efferent data sources 306 may include video eyetracking data 312. Data 312 may measure gaze direction, pupil size,blinks, and any other data pertaining to user's 302 eyes that may bemeasured using a Video Eye Tracker (VET) or an electro-oculogram. Thisis also illustrated in FIG. 8 , which shows characteristic metrics 800for eye tracking, in accordance with an exemplary embodiment of thepresent specification. Video eye tracking 802 generally involvesrecording images of a user's eye(s) and using image processing toidentify the pupil and specific reflections of known light sources(typically infrared) from which may be derived measures of pupil sizeand gaze direction. The angular resolution (of eye gaze direction) andtemporal resolution (frames per second) may limit the availability ofsome measures. Some measures may be recorded as discrete events, andothers recorded over time for analysis of trends and statistics overepochs of time.

Gaze Direction

Software, typically provided with the eye tracking hardware, may providecalibrated estimates of gaze direction in coordinates tied to thedisplay used for calibration. It may be possible/necessary to performsome of this conversion separately. For head mounted units with externalview cameras the gaze position may be in head centric coordinates or incoordinates relative to specific objects (perhaps provided referenceobjects) in the environment. It is assumed that gaze direction will beprovided at some rate in samples per second. Most of the followingmetrics will be derived from this stream of gaze direction data:saccade, pursuit, vergence, patterns, and/or microsaccades.

Saccade: Prolonged periods of relatively fixed gaze direction separatedby rapid changes in gaze (over a matter of milliseconds) may be loggedas “fixations” and the jumps in between as “saccades”. Fixations will benoted for position, start and end time and duration. In some cases theymay also be rated for stability (variability of gaze direction duringfixation). Saccades will be noted for their direction (angle), speed anddistance. It is worth noting, and it will generally be assumed, thatthere is a period of cortical suppression during saccades when visualinformation is not (fully) processed. This saccadic suppression may beexploited by developers to alter displays without creating a percept ofmotion, appearance or disappearance among display elements.

Pursuit: Pursuit eye movements may be characterized by smooth changes ingaze direction, slower than typical saccades (and without corticalsuppression of visual processing). These smooth eye movements generallyoccur when the eyes are pursuing/tracking an object moving relative tohead facing direction, a stationary object while the head moves ormoving objects while the head also moves. Body or reference frame motioncan also generate pursuit eye movements to track objects. Pursuit canoccur in the absence of a visual stimulus based on the anticipatedposition of an invisible or obscured target.

Vergence: This measure may require relatively fine resolution gazedirection data for both eyes simultaneously so that the difference ingaze direction between eyes can be used to determine a depth coordinatefor gaze. Vergence is in relation to the distance of the object in termsof the user to measure objects between the near point of convergence andtowards infinity in the distance—all of which may be modelled based offthe measurements of vergence between convergence and divergence.

Patterns: Repeated patterns of eye movements, which may be derived frommachine learning analysis of eye gaze direction data, may be used tocharacterize response events, states of user interaction or to measureeffects of adaptation, training or learning. Notable are patterns duringvisual search for targets or free viewing of scenes towards thecompletion of a task (e.g. learning of scene details for laterrecognition in a memory task). Eye movement patterns may also be used togenerate models for creating saliency maps of scenes, guiding imageprocessing.

Microsaccades: With relatively sensitive direction and time resolutionit may be possible to measure and characterize microsaccadic activity.Microsaccades are generally present during fixation, and are ofparticular interest during rigid or prolonged fixation. Feedback into adisplay system may allow for creating images that remain static on theretina resulting in Troxler fading. Microsaccades are not subject toconscious control or awareness.

Sample questions concerning eye tracking metrics that may be answeredover a period of time may include: where are users looking the most(potentially in response to repeating events), how fast and accurate aresaccadic eye movements, how rapidly are users finding targets, are userscorrectly identifying targets, how accurate is pursuit/tracking, arethere preferences for certain areas/stimuli.

During free viewing or search, fixations (relatively stable eye gazedirection) between saccades typically last on the order of 200-300milliseconds. Saccades have a rapidly accelerating velocity, up to ashigh as 500 degrees per second, ending with a rapid deceleration.Pursuit eye movements occur in order to steadily fixate on a movingobject, either from object motion or head motion relative to the objector both. Vergence eye movements are used to bring the eyes together tofocus on near objects. Vestibular eye movements are compensatory eyemovements derived from head and/or body movement.

Reference is made to WO2015003097A1 entitled “A Non-Invasive Method forAssessing and Monitoring Brain”. In an example, a pro-saccade eyetracking test is performed. The pro-saccade test measures the amount oftime required for an individual to shift his or her gaze from astationary object towards a flashed target. The pro-saccade eye trackingtest may be conducted as described in The Antisaccade: A Review of BasicResearch and Clinical Studies, by S. Everling and B. Fischer,Neuropsychologia Volume 36, Issue 9, 1 Sep. 1998, pages 885-899(“Everling”), for example.

The pro-saccade test may be performed while presenting the individualwith a standardized set of visual stimuli. In some embodiments, thepro-saccade test may be conducted multiple times with the same ordifferent stimuli to obtain an average result. The results of thepro-saccade test may comprise, for example, the pro-saccade reactiontime. The pro-saccade reaction time is the latency of initiation of avoluntary saccade, with normal values falling between roughly 200-250ms. Pro-saccade reaction times may be further sub-grouped into: ExpressPro-Saccades: 80-134 ms; Fast regular: 135-175 ms; Slow regular: 180-399ms; and Late: (400-699 ms).

Similarly, an anti-saccade eye tracking test may be performed. Theanti-saccade test measures the amount of time required for an individualto shift his or her gaze from a stationary object away from a flashedtarget, towards a desired focus point. The anti-saccade eye trackingtest can be conducted as described in Everling, for example. In someexamples, the anti-saccade test may also measure an error time and/orerror distance; that is, the amount of time or distance in which the eyemoves in the wrong direction (towards the flashed target). Theanti-saccade test may be performed using the standardized set of visualstimuli. The results of the anti-saccade test may comprise, for example,mean reaction times as described above for the pro-saccade test, withtypical mean reaction times falling into the range of roughly 190 to 270ms. Other results may include initial direction of eye motion, final eyeresting position, time to final resting position, initial fovea distance(i.e., how far the fovea moves in the direction of the flashed target),final fovea resting position, and final fovea distance (i.e., how farthe fovea moves in the direction of the desired focus point).

Also, a smooth pursuit test may be performed. The smooth pursuit testevaluates an individual's ability to smoothly track moving visualstimuli. The smooth pursuit test can be conducted by asking theindividual to visually follow a target as it moves across the screen.The smooth pursuit test may be performed using the standardized set ofvisual stimuli, and may be conducted multiple times with the same ordifferent stimuli to obtain an average result. In some embodiments, thesmooth pursuit test may include tests based on the use of fade-in,fade-out visual stimuli, in which the target fades in and fades out asthe individual is tracking the target. Data gathered during the smoothpursuit test may comprise, for example, an initial response latency anda number of samples that capture the fovea position along the directionof motion during target tracking. Each sampled fovea position may becompared to the position of the center of the target at the same time togenerate an error value for each sample.

For more sensitive tracking hardware, it may also be possible to measurenystagmus (constant tremor of the eyes), drifts (due to imperfectcontrol) and microsaccades (corrections for drift). These will alsocontribute noise to gross measurements of gaze position; as a resultfixations are often characterized by the mean position over a span ofrelatively stable gaze position measures. Alternatively, a threshold ofgaze velocity (degrees/second) can be set, below which any smallmovements are considered to be within a fixation.

Saccades require time to plan and execute, and a delay, or latency, ofat least 150 ms is typical after, for example, the onset of a visualstimulus eliciting the saccade. Much can be said about the latencybefore a saccade and various contexts that may lengthen or shorten them.The more accurate information we have regarding the relative timing ofeye movements and events occurring in the visual scene, the more we cansay about the effect of stimulus parameters on saccades.

Although usually correlated, shifts in attention and eye gaze do notnecessarily have to happen together. In some contexts it may beefficient for the user to direct attention to a point in their visualperiphery, for example to monitor one location while observing another.These scenarios may be useful for generating measures related to Fieldof View and Multi-Tracking.

It is possible to use image processing techniques to highlight regionswithin a scene of greater saliency based on models of the visual system.For example areas of greater high-spatial-frequency contrast (i.e. edgesand lines) tend to capture attention and fixations. It is possiblewithin a specific context to use eye gaze direction to develop customsaliency maps based on the information available in the visual scenecombined with whatever tasks in which an observer may be engaged. Thistool can be used to highlight areas of interest or greater engagement.

Pupil Size

Pupil size may be measured as part of the image processing necessary toderive gaze direction. Pupil size may generally change in response tolight levels and also in response to certain stimulus events viaautonomic process. Pupil responses are not subject to conscious controlor awareness (except secondarily in the case of extreme illuminationchanges). Sample questions concerning eye tracking metrics that may beanswered over a period of time may include: how are the pupilsresponding to different stimuli, how are the pupils behaving over time.

Pupil diameter generally falls between 2 and 8 mm at the extremes inlight and dark, respectively. The pupil dilates and constricts inresponse to various internal and external stimuli. Due to differences inbaseline pupil diameter, both among observers and due to ambientlighting and physiological state, pupil responses may generally bemeasured as proportions of change from baseline. For example, thebaseline pupil diameter might be the diameter at the moment of anexternal stimulus event (image appears), and the response is measured bythe extent to which the pupil dilates or constricts during the 1 secondafter the stimulus event. Eye color may affect the extent ofconstriction, and age may also be a factor.

In addition to responding to light, accommodation for distance and otherspatial and motion cues, pupil diameter will often be modulated bycognitive load, certain imagery and reading. Pupil diameter may bemodulated during or at the termination visual search. Proportionalchanges can range from a few to tens of percentage points.

Thresholds for determining computationally if a response has been madewill vary depending on the context and on the sensitivity of thehardware used. Variations in ambient lighting and/or the mean luminanceof displays will also have a large influence on pupil diameter andproportional changes, so thresholds will need to be adaptable and likelydetermined by the data itself (e.g. threshold for dilation event itselfbeing a percentage of the range of pupil diameter values recorded withina session for one user).

Reference is again made to WO2015003097A1 titled “A Non-Invasive Methodfor Assessing and Monitoring Brain”. In an example, pupillary responseis assessed. Pupillary response is often assessed by shining a brightlight into the individual's eye and assessing the response. In fieldsettings, where lighting is difficult to control, pupillary response maybe assessed using a standardized set of photographs, such as theInternational Affective Picture System (IAPS) standards. Thesephotographs have been determined to elicit predictable arousal patterns,including pupil dilation. The pupillary response test may be performedusing a variety of stimuli, such as changes to lighting conditions(including shining a light in the individual's eyes), or presentation ofphotographs, videos, or other types of visual data. In some embodiments,the pupillary test may be conducted multiple times with the same ordifferent stimuli to obtain an average result. The pupillary responsetest may be conducted by taking an initial reading of the individual'spupil diameter, pupil height, and/or pupil width, then presenting theindividual with visual stimuli to elicit a pupillary response. Thechange in pupil dilation (e.g., the change in diameter, height, width,and/or an area calculated based on some or all of these measurements)and the time required to dilate are measured. The results of thepupillary response test may include, for example, a set of dilation(mydriasis) results and a set of contraction (miosis) results, whereeach set may include amplitude, velocity (speed ofdilation/constriction), pupil diameter, pupil height, pupil width, anddelay to onset of response.

Blinks

Video eye trackers, as well as less specialized video imaging of auser's face/eye region, may detect rapid or prolonged periods of eyeclosure. Precautions may be taken as loss of acquisition may also be acause for periods of data loss. Blink events, conscious or reflexive,and blink rates over time related to measures of fatigue or irritationmay be recorded. Sample questions concerning eye tracking metrics arementioned in FIG. 8 . In embodiments, these are questions that may beanswered over a period of time and may include: are the users blinkingin response to the onset of stimuli, is the blink rate changing inresponse to the stimuli, is the blink rate changing overall, does theblink rate suggest fatigue.

Normal blinking rates among adults are around 10 blinks per minute atrest, and generally decreases to around 3 blinks per minute duringfocused attention (e.g. reading). Other properties of blinks, forexample distance/speed of eyelid movement and durations of variousstages within a blink, have been correlated with error rates innon-visual tasks (for example, using auditory stimulus discrimination)and other measures; whenever possible it may be advantageous to usevideo recordings to analyze eyelid position in detail (i.e. automatedeyelid tracking). Blink durations longer than 150 ms may be consideredlong-duration blinks.

As with most measures, proportional changes from baseline may be morevaluable than absolute measures of blink frequency or average duration.Generally, significance can be assigned based on statistical measures,meaning any deviation is significant if it is larger than the generalvariability of the measure (for example as estimated using a t-test).

Manual Inputs

Referring back to FIG. 3 , another efferent data source 306 may bemanual input 314. Which have been a traditional tool of computerinteraction and may be available in many forms. Exemplary manual inputs314 of interest include input identity (key pressed), any other gesture,position coordinates (x, y, z) on a touch screen or by a mouse, and/or(video) tracking of hand or other limb. FIG. 9 illustratescharacteristic metrics 900 for manual inputs 902, in accordance with anembodiment of the present specification.

Sample questions concerning manual input metrics that may be answeredover a period of time may include: where are the users clicking/touchingthe most (potentially in response to repeating events), how fast andaccurate are the clicks/touches, how rapidly are users finding targets,are users correctly identifying targets, how accurate is tracking, arethere preferences for certain areas/stimuli, what kind ofgrasping/touching motions are the users making, how is the hand/eyecoordination, are there reflexive actions to virtual stimuli.

Responses made with the fingers, hands and/or arms, legs, or any otherpart of the body of users may generally yield timing, position,trajectory, pressure and categorical data. These responses may bediscrete in time, however some sustained or state variable may be drawnfrom manual data as well. Following analytic response metrics may bederived from manual responses: category, identity, timing, position,and/or trajectory.

Category: In addition to categories like click, touch, drag, swipe andscroll there may be sub categories like double click, tap or push,multi-finger input, etc. Any variable that differentiates one actionfrom another by category that is detectable by an application may beimportant for differentiating responses (and will likely be used forthat purpose by developers).

Identity: Whenever multiple input modalities exist for the same type ofresponse event, most notably the keys on a computer keyboard, or anyother gesture that may be possible in a media environment, the identityof the input may be recorded. This also includes directions indicated ona direction pad, mouse buttons clicked and, when possible, the area of atouchpad touched (independent of cursor position), or any other gesture.

Timing: The initiation and ending time of all responses may be recorded(e.g. a button press will log both the button-down event and thebutton-up event), and from that response durations can be derived. Thistiming information will be key to connecting responses to the stimulithat elicited them and correlating events in time.

Position: For visual interfaces, the position may be in displaycoordinates. Positions may be singular for discrete events like clicksor continuously recorded at some reasonable rate for tracing, dragging,etc. When possible these may also be converted to retinal coordinates(with the combination of eye gaze tracking). By understanding position,a topography of the retina may be done, and areas of the retina may bemapped in relationship to their specific functions further inrelationship to the brain, body, endocrine, and autonomic systems. Forgestures recorded by video/motion capture the body-centric position willbe recorded along with the location of any cursor or other object beingcontrolled by the user.

Trajectory: For swipe, scroll and other dynamic gestures it may bepossible to record the trajectory of the response (i.e. the directionand speed as a vector) in addition to any explicit position changes thatoccur. This will, in fact, likely be derived from an analysis of rapidchanges in position data, unless the device also provides event typesfor these actions.

Head Tracking

Head tracking measures are largely associated with virtual, augmented,and mixed reality displays. They can provide measures of synchrony withdisplayed visual environments, but also of users' reactions to thoseenvironments. Orienting towards or away from stimuli, compensatorymovements in line or not in line with the displayed visual environmentsand other motion behavior can be used to derive similar, though lessprecise, measures similar to those from eye tracking. Those derivedmeasures associated with arousal, fatigue and engagement can be modifiedas previously stated.

If head movements, particularly saccadic head movements, prove to be asource of mismatch and discomfort for users it may be desirable tomodify displays to reduce the number of such head movements. Keepingdisplay elements within a region near head-center and/or encouragingslower changes in head-facing may reduce large head movements. Withregards to individual differences: some users will move their heads morethan others for the same scenario. It may be possible to train headmovers to reduce their movements.

Referring back to FIG. 3 , head tracking data 316 may be another form ofefferent data 306 source. Head tracking data 316 may track user's 302head orientation and physical position from either video tracking (VETor otherwise) or position sensors located on HMDs, headsets, or otherworn devices. In addition to tracking user's 302 head, their body may betracked. The position of users' 302 bodies and parts thereof may berecorded, likely from video based motion capture or accelerometers inwearable devices. This position data would commonly be used to encodemanual response data (coming from finger, hand or arm tracking) and/orhead orientation relative to the environment to aid in eye gazemeasurements and updating of the user's visual environment. Headposition data may also be used to model the effect of head shadow onsounds coming from the environment. FIG. 10 illustrates characteristicmetrics 1000 for head tracking, which may include head orientation 1002and/or physical position 1004, in accordance with an embodiment of thepresent specification.

Sample questions concerning head tracking metrics that may be answeredover a period of time may include: where are the users looking most(potentially in response to repeating events), how fast and accurate arehead movements, how accurate is pursuit/tracking, is there preferencefor certain areas/stimuli, are users accurately coordinating head andeye movements to direct gaze and/or track objects, are head movementsreduced due to the hardware, are users making many adjustments to thehardware, are users measurably fatigued by the hardware.

Head movements may be specifically important in the realms of virtual,augmented, and mixed reality, and may generally be correlated with eyemovements, depending upon the task. There is large individualvariability in propensity for head movements accompanying eye movements.During tasks like reading, head movement can account for 5% to 40% ofshifting gaze (combined with eye movements). The degree to which a usernormally moves their head may prove a key indicator of susceptibility tosickness from mismatch of visual and vestibular sensation.

It is likely that saccadic and pursuit head movements may bequalitatively different in those two modalities. For example, a mismatchmay be less jarring if users follow an object from body front, 90degrees to the right, to body side using a pursuit movement as opposedto freely directing gaze from forward to the right. If the velocity of apursuit object is relatively steady then the mismatch would beimperceptible through most of the motion.

Referring back to FIG. 3 , a user's 302 vocal responses may also betracked via microphone. Speech recognition algorithms would extractsemantic meaning from recorded sound and mark the time of responses(potentially of individual words or syllables). In less sophisticatedscenarios the intensity of vocal responses may be sufficient to mark thetime of response. In embodiments, voice and speech data is correlatedwith several other forms of data such as and not limited to headtracking, eye-tracking, manual inputs, in order to determine levels ofperception.

Electrophysiology/Autonomous Recording

Electrophysiological and autonomic measures fall largely outside therealm of conscious influence and, therefore, performance. These measurespertain largely to states of arousal and may therefore be used to guidestimulus selection. Recounted for convenience here, the measures ofinterest would come from electroencephalography (EEG—specifically theactivity of various frequency bands associated with arousal states),galvanic skin response (GSR—also associated with arousal and reaction toemotional stimuli), heart rate, respiratory rate, blood oxygenation, andpotentially measures of skeletal muscle responses.

Reference is again made to WO2015003097A1 titled “A Non-Invasive Methodfor Assessing and Monitoring Brain”. In an example, brain wave activityis assessed by performing an active brain wave test. The active brainwave test may be conducted using EEG (electroencephalography) equipmentand following methods known in the art. The active brain wave test maybe performed while the individual is presented with a variety of visualstimuli. In some embodiments, the active brain wave test is conductedwhile presenting a standardized set of visual stimuli that isappropriate for assessing active brain wave activity. In someembodiments, the active brain wave test may be conducted multiple times,using the same or different visual stimuli, to obtain an average result.The results of the active brain wave test may comprise, for example,temporal and spatial measurements of alpha waves, beta waves, deltawaves, and theta waves. In some embodiments, the results of the activebrain wave test may comprise a ratio of two types of brain waves; forexample, the results may include a ratio of alpha/theta waves.

Similarly, a passive brain wave test may be performed. The passive brainwave test may be conducted using EEG (electroencephalography) equipmentto record brain wave data while the individual has closed eyes; i.e., inthe absence of visual stimuli. The results of the passive wave brainwave test may comprise, for example, temporal and spatial measurementsof alpha waves, beta waves, delta waves, and theta waves, for example.In some embodiments, the results of the passive brain wave test maycomprise a ratio of two types of brain waves; for example, the resultsmay include a ratio of alpha/theta waves. In some embodiments, thepassive brain wave test may be conducted multiple times to obtain anaverage result.

When possible, and reliant upon precise timing information for bothelectric potentials and stimulus displays/speakers, time-averagedresponses can be generated from repeated trials. Characteristicwaveforms associated with visual or auditory processing (Event RelatedPotentials, ERP) can be measured and manipulated in various ways. Asthese do not require volitional behavior from users they represent alower-level, arguably more pure measure of perception.

Referring back to FIG. 3 , electrophysiological data 318 may be yetanother efferent data source 306, which may generally be available inthe form of voltage potentials recorded at a rate on the order of kHz.This may include any and all measurements of voltage potentials amongelectrodes placed on the skin or other exposed tissue (notably thecornea of the eye). Most use cases would presumably involve noninvasiverecording, however opportunities may arise to analyze data fromimplanted electrodes placed for other medically valid purposes. Data maygenerally be collected at rates in the hundreds or thousands of samplesper second. Analyses may focus on either time-locked averages ofresponses to stimulus events to generate waveforms or on variousfiltered representations of the data over time from which various statesof activity may be inferred. For example, Electroencephalogram (EEG) maybe used to gather electrode recording from the scalp/head, to revealelectrical activity of the brain and other neural activity. Recordingmay focus on areas of primary sensory processing, secondary and latersensory processing, cognitive processing or response generation (motorprocessing, language processing). An Electrooculogram (EOG) may beutilized to gather electrode recording from near the eye to measurechanges in field potential due to relative eye position (gaze direction)and can also measure properties of retinal function and muscle activity.EOG may provide a low spatial resolution substitute for video eyetracking. An Electroretinogram (ERG) may be used to gather electroderecording from the cornea (minimally invasive) to capture neuralactivity from the retina. Correlation with chromatic and spatialproperties of stimuli may allow for the characterization of responsesfrom different cone types and locations on the retina (this is also thecase with visual evoked potentials recorded via EEG). AnElectrocardiogram (ECG) may be used to gather neuromuscular activitycorresponding to cardiac function and provide measures of autonomicstates, potentially in response to stimuli. Measurement of neuromuscularpotentials may involve electrodes placed anywhere to recordneuromuscular activity from skeletal muscle flex and/or movement of bodyand limb (including electromyogram, or EMG). Measurement of GalvanicSkin Response (GSR) may involve electrodes that can measure potentialdifferences across the skin which are subject to conductance variationsdue to sweat and other state changes of the skin. These changes areinvoluntary and may reveal autonomic responses to stimuli or scenarios.

Another source of efferent data 306 may be autonomic monitoring data320, including information about heart rate, respiratory rate, bloodoxygenation, skin conductance, and other autonomic (unconscious)response data from user 302 in forms similar to those forelectrophysiological data 318. Pressure transducers or other sensors mayrelay data about respiration rate. Pulse oximetry can measure bloodoxygenation. Pressure transducers or other sensors can also measureblood pressure. Any and all unconscious, autonomic measures may revealresponses to stimuli or general states for categorization of other data.FIG. 11 illustrates characteristic metrics 1100 for electrophysiologicalmonitoring data 1102 and autonomic monitoring data 1104, in accordancewith an embodiment of the present specification.

Sample questions concerning electrophysiological metrics 1102 andautonomic metrics 1104 that may be answered over a period of time mayinclude: what are the characteristics of time-averaged responses toevents, how do various frequency bands or other derived states changeover time or in response to stimuli.

Sensors for collecting data may be a part of hardware 106, describedabove in context of FIG. 1 . Some sensors can be integrated into an HMD(for example, sensors for electroencephalography, electrooculography,electroretinography, cardiovascular monitoring, galvanic skin response,and others). Referring back to FIG. 3 , some data may require sensorselsewhere on the body of user 302. Non-contact sensors (even video) maybe able to monitor some electrophysiological data 318 and autonomicmonitoring data 320. In embodiments, these sensors could be smartclothing and other apparel. It may be possible to use imaging data forusers, to categorize users or their present state. Functional imagingmay also provide data relating to unconscious responses to stimuli.Imaging modalities include X-Ray/Computed Tomography (CT), MagneticResonance Imaging (MRI), Ophthalmic Imaging, Ultrasound, andMagnetoencephalography (MEG). Structural data derived from imaging maybe used to localize sources of electrophysiological data (e.g. combiningone or more of structural, MRI EEG, and MEG data).

Metrics may be broken into direct measures that can be inferred fromthese stimulus/response feature pairs, and indirect measures that can beinferred from the direct measures. It should be understood that in mostcases individual occurrences of stimulus/response feature pairings maybe combined statistically to estimate central tendency and variability.There is potential value in data from a single trial, from descriptivestatistics derived from multiple repeated trials of a particulardescription and from exploring stimulus and/or response features ascontinuous variables for modelling and prediction.

Facial Pattern Recognition Machine Learning

The SDEP may utilize its models and predictive components in combinationwith a product to enable development of a customized predictivecomponent for the product. The SDEP predictive components may be builtthrough a collection process by which a large dataset of vision datafrom naturalistic or unconstrained settings from both primary andsecondary sources may be curated and labeled. The dataset may includephotographs, YouTube videos, Twitch, Instagram, and facial datasets thatare available through secondary research, such as through the Internet.The curated and labeled data may be utilized for further engagement, andto build a custom-platform for the product.

FIGS. 12A to 12D illustrate an exemplary process of image analysis ofbuilding curated data. The illustrations describe an exemplarymobile-based version of the model. In other embodiments, the model maybe executed on the cloud. FIG. 12A illustrates an exemplary image of asubject for whom a customized predictive component may be developed.FIG. 12B illustrates an image of the subject where the SDEP identifiesthe eyes for eye tracking, blink detection, gaze direction, and otherparameters and/or facial attributes. In embodiments, the eyes arecontinually identified for tracking purposes through a series of imagesor through a video of the subject.

FIG. 12C illustrates a dataset 1202 of vision data from naturalistic orunconstrained settings, which may be used for extracting face attributesin the context of eye tracking, blink, and gaze direction. Inembodiments, the SDEP system is trained with a large data set 1202 underdifferent conditions where the frames are extracted from videos.Different conditions may include among other, complex face variations,lighting conditions, occlusions, and general hardware used. Inembodiments, various computer vision techniques and Deep Learning areused to train the system. Referring to FIGS. 12C and 12D, image 1204 isselected to extract its face attributes for analyzing emotions of thesubject. In embodiments, images from the dataset, including image 1204,are curated and labelled.

The following steps outline an exemplary data curation and labellingprocess:

-   1. Identify desirable data sources-   2. Concurrently, develop a pipeline to perform facial key point    detection from video and still images. This may be achieved by    leveraging facial key point localization to segment and select the    ocular region from faces. Further key point features may be used to    determine rotation, pitch, and lighting of images, as possible    dimensions to marginalize over in downstream analysis. Facial    expressions may be identified to analyze emotions. Blinks, eye    movements, and microsaccades may also be identified as part of the    key point detection system.-   3. Scrapes of data sources may be identified and fed through the    SDEP (see FIG. 2B) to obtain a normalized set of ocular region    images. Final images may be segmented/cropped to include only the    ocular region, such that information on pitch, rotation, and    lighting is available upon return.-   4. Output from the above processing may be combined with a product    to label blink, coloration, strabismus, and other metrics of    interest to the product.

The above-mentioned collected and labelled data may be leveraged todevelop custom predictive models of the ocular region. Customizedmachine learning algorithms may be created to predict key parametersranging from blink rate, fatigue, emotions, gaze direction, attention,phorias, convergence, divergence, fixation, gaze direction, pupil size,and others. In addition, multimodal approaches may leverage the SDEP inorder to benefit from pixel level information in digital stimuli andjointly learn relationships with ocular response. The pixel levelinformation may be broken down to RGB, luminance to fuse the same withexisting visual modeling algorithms.

In embodiments, eye tracking parameters are extracted from eye trackingalgorithms. In an embodiment, pupil position, relative to the face,provides one measure from which to classify eye movements as fixations,pursuits and saccades. In an embodiment, pupil size is also measured,independently for both eyes. In an embodiment, gaze direction isestimated from relative pupil position. Gaze position may be measured in3D space using data from both eyes and other measures (i.e. relativeposition of the face and screen), including estimates of vergence. GazePosition provides another measure from which to classify eye movements.

FIGS. 13A and 13B illustrate pupil position and size and gaze positionover time. While FIG. 13A illustrates pupil position and size and gazeposition in 3D 1304A and 2D 1310A, at a first time; FIG. 13B illustratespupil position and size and gaze position in 3D 1304B and 2D 1310A, at asecond time. In an embodiment the second time is later than the firsttime. At any given point in the image there is (up to) 1 second of databeing shown, with older data shown in a different color, such as blue.The light blue square represents the display at which the observer waslooking. Physical dimensions are not to scale (e.g. the viewing distancewas greater than it appears to be in the left panel). The left panel1304A and 1304B shows a 3D isometric view of space with user's eyes 1306to the left and the display 1308 to the right.

On the left side, gaze position is shown in 3D 1304A and 1304B. A lineis drawn from the surface of the observer's display 1308 to the gazeposition; red indicates gaze position behind the display 1308 and greenindicates gaze position in front of the display 1308. Three circlesconvey information about the eyes 1306:

-   -   1. The largest, dark grey outline circle represents the average        position of the eyes and face, relatively fixed in space.    -   2. The light grey outline within represents the average pupil        size and pupil position relative to the face (moves but doesn't        change size).    -   3. The black filled circle shows relative pupil size as well as        pupil position relative to the face (moves and changes size).

When the pupil information is missing it may be assumed that the eyesare closed (or otherwise obscured).

Gaze position in 3D 1304A and 1304A is shown by a black dot (connectedby black lines), with gaze direction emanating from both eyes. Depth ofgaze from the display is further indicated by a green (front) or red(behind) line from the display to the current gaze position.

On the right side, gaze position 1310B and 1310B is shown in 2D. Hereinformation about the pupils is absent. Also, information classifyingeye movements is added:

-   -   1. Black indicates fixation during which a grey outline grows        indicating relative duration of the fixation.    -   2. Blue indicates pursuit.    -   3. Green (with connecting lines) indicates saccades with lines        connecting points during the saccade.        Vision Performance Index

An important class of metrics may be those relating to performance. Theperformance of a user may be determined in the form of VisionPerformance Index (VPI), which is described in detail subsequently inembodiments of the present specification.

Referring back to FIG. 1 , in an embodiment, data collected from user102, such as by the media system 104, may be processed to identify aVision Performance Index (VPI) for user 102 (also referring to 240 ofFIG. 2B). The VPI may indicate a level of vision performance of user 102assessed during user's 102 interaction with the media system 104. TheVPI may be used to identify a group of users for user 102 that have asimilar VPI.

VPI may be measured and manipulated in various ways. In general, thegoal may be to improve user's vision performance, however manipulationsmay also be aimed at increasing challenge (e.g. for the sake ofengagement) which may, at least temporarily, decrease performance. Inalternate embodiments, performance indices other than or in addition tothat related to vision may be measured and manipulated. For example,other areas such as design, engagement, and the like, may be measuredand manipulated through performance indices.

Referring again to FIG. 2B, an exemplary outline of a data analysischain is illustrated. The data analysis begins at the lowest level at232 where data level may not be simplified further. At 232, parametersof a single stimulus can be used for multiple measures based ondifferent independent variables, which correspond to direct features ofa stimulus. Parameters of a single response can be used for multiplemeasures based on different dependent variables. At 234 independent anddependent variables may be paired to extract a measure of a user'svision performance, or combined with others and fit to a model togenerate measures of the user's vision performance. In embodiments,pairing may involve combining a response event to one or more stimulusevents through correlation or other statistical/non-statistical methods.Individual pairs may be filtered to arrive at 236, where, for a giventype of interaction, many pairs of independent and dependent variablescan be used to either estimate the parameters of a model distribution orestimate descriptive statistics. In embodiments, a model distribution isan expectation of how often a measure will be a specific value. In someinstances a normal distribution, which has the classic shape of a ‘Bellcurve’, may be used. Once the process of descriptive statistics or modelfitting is completed, at 238, an individual estimate of a physicalmeasure of a property of user's vision may be generated. The individualuser estimate may be based on a single interaction or a summary measurefrom multiple interactions. The measures of at least one physicalproperty may be normalized to contribute to sub-components of VPI, at240. At 242, multiple VPI sub-components scores may be combined (forexample, averaged) to generate component scores. In embodiments,component scores may be further combined to generate overall VPI. VPI,its subcomponents, and components are discussed in greater detail insubsequent sections of the present specification.

In embodiments, measures of vision performance may be presented as anormalized “score” with relative, but not absolute, meaning, to theusers. This is also illustrated at 240 and 242 in context of FIG. 2B.Users may be able to gauge their level of performance against thegeneral population, or specific subsets thereof. Due to the presumedhigh degree of measurement noise associated with data recording fromnon-specialized hardware (i.e. mobile devices used outside of acontrolled experimental setting), precise measures of efferent phenomena(e.g. pupil size, gaze direction, blink detection) and afferentparameters (e.g. display chromoluminance, viewing distance, audiointensity) are unavailable. It may therefore be required to rely onestimates of central tendency (i.e. mean) and variability (i.e. standarddeviation) from the accumulated data of all users to define “typical”ranges for each measure and to set reasonable goals for increasing ordecreasing those measures.

Scores may be normalized independently for each type of measure, foreach of a variety of types of tasks and generally for each uniquescenario or context. This may enable easy comparison and averagingacross measures taken in different units, to different stimuli, and fromdifferent kinds of user responses. Additionally, for any and all scores,performance may be categorized as being marginally or significantlyabove or below average. Set descriptive criteria may be decided based onpercentiles (assuming a given measure will be distributed normally amongthe general population). The examples in the following sections use 10%and 90%, however the percentiles may be arbitrarily chosen and can bemodified for specific contexts. It may be assumed that 10% of users'scores will fall in the bottom or top 10% of scores, and therefore be‘abnormally’ low or high, respectively.

In an embodiment, VPI may be a combination of one or more of thefollowing parameters and sub-parameters, which may be both afferent andefferent in nature. In some embodiments, the VPI may be a function ofpsychometric data, without efferent data. Direct measures generallyrelate a single response feature to a single stimulus feature. Wheneverpossible a psychometric function may be built up from the pattern ofresponses (average response, probability of response or proportion of acategory of responses) as the stimulus feature value changes. Directmeasure may include the following: detection, discrimination, responsetime, and/or error.

Indirect measures may be the higher level interpretations of the directmeasures and/or combinations of direct measures. These may alsogenerally include descriptions of direct measures within or acrossspecific contexts and the interactions among variables. Indirectmeasures may include the following: multi-tracking, fatigue/endurance,adaptation/learning, preference, memory, and/or states.

In embodiments, other vision-related parameters may be used to calculatethe VPI, and may include, but are not limited to field of view (F),accuracy (A), multi-tracking (M), endurance (E), and/ordetection/discrimination (D), together abbreviated as FAMED, alldescribed in greater detail below.

Field of View (F)

Referring back to FIG. 1 , the Field of View (F) may be described as theextent of visual world seen by user 102 at any given moment. Centralvision represents a central part of the field of view of user 102, whereuser 102 has the greatest acuity which is important for things likereading. Peripheral Vision is the external part of the field of view ofuser 102, which is important for guiding future behavior and catchingimportant events outside of user's 102 focus.

Field of View measures the relative performance of users wheninteracting with stimuli that are in their Central or Peripheral fieldsof view based on measures of Accuracy and Detection. It is assumed thatperformance should generally be worse in the periphery due to decreasedsensitivity to most stimulus features as visual eccentricity increases.The ratio of performance with Central and Peripheral stimuli will havesome mean and standard deviation among the general population; as withother measures, the normalized scores will be used to determine if usershave abnormally low or high Field of View ability.

If a user's Field of View score is abnormally low it may be improved byincreasing the Accuracy and Detection scores for stimuli presented inthe periphery. This generally would entail increasing consistency oftiming and position, increasing chromaticity and luminance differences(between and within objects), increasing the size of objects and slowingany moving targets when presented in the periphery.

Accuracy (A)

Referring back to FIG. 1 , accuracy (A) may be a combination of makingthe right choices and being precise in actions performed by user 102.Measures of accuracy may be divided into two subcomponents: Reaction andTargeting. Reaction relates to the time it takes to process and act uponincoming information. Reaction may refer to ability of the user 102 tomake speedy responses during the media experience. Reaction may bemeasured as the span of time between the point when enough informationis available in the stimulus to make a decision (i.e. the appearance ofa stimulus) and the time when the user's response is recorded. For aspeeded response this will usually be less than one second.

If a user's Reaction is abnormally slow (abnormally low score) it may bethat the task is too difficult and requires modification of stimulusparameters discussed later in the context of Targeting and Detection. Inan embodiment, a model distribution for any given measure (for example,a log-normal distribution for reaction times) is estimated. A cut-offmay be determined from the estimate, above which 5% (or any otherpercentage) slowest time spans are found. Any incoming measure ofreaction time that is equal or greater to the cut-off is considered‘slow’ (or ‘significantly slow’). However, if reaction alone isabnormally low, when other scores are normal, it may be a sign of poorengagement with the task or a distraction. It may be helpful to reducethe number of items presented simultaneously or add additional,congruent cues to hold attention (e.g. add a sound to accompany theappearance of visual stimuli). If the user is required to respond to thelocation of a moving object, it may be that they require longer toestimate trajectories and plan an intercepting response; slowing of thetarget may improve reaction.

Response Time may be important for detection related measures, but isrelevant to any response to a stimulus. Response time is generally thetime span between a stimulus event and the response to that event.Response time may be used to measure the time necessary for the brain toprocess information. As an example, the appearance of a pattern on adisplay may lead to a certain pattern of responding from the retinameasurable by ERG. At some point after the stimulus processing isevident from an averaged ERG waveform, the processing of that samestimulus will become evident in an average visual evoked potential (VEP)waveform recorded from the back of the head. At some point after thatthe average time to a button press response from the user indicates thatthe stimulus was fully processed to the point of generating a motorresponse. Though multiple timestamps may be generated by stimulus andresponse events, the response time should generally be taken as the timebetween the earliest detectable change in the stimulus necessary tochoose the appropriate response to the earliest indication that aresponse has been chosen. For example, if an object begins moving in astraight line towards some key point on the display, that initial bit ofmotion in a particular direction may be enough for the user to knowwhere the object will end up. They need not wait for it to get there.Likewise the initiation of moving of the mouse cursor (or any othergesture acceptable in a VR/AR/M×R environment) towards a target to beclicked may indicate that a response has been chosen, well before theclick event actually occurs.

In embodiments, other changes in patterns of responding, includingimprovements, decrements and general shifts, may occur as the result ofperceptual adaptation, perceptual learning and training (higher orderlearning). Considering adaptation and learning by the user may accountfor any variability in responses that can be explained, and therebyreduce measures of statistical noise and improve inferential power.

Patterns in responding, and changes thereof, may also be related to highorder processes within the system. Users have an occasional tendency tochange their minds about how they perform a task while they're doing it.Therefore, in embodiments, every choice made by users is analyzed forpreferences, regardless of whether it informs models of visualprocessing.

In embodiments, responses are used by the system to measure recall orrecognition by a user. Recall is the accurate generation of informationpreviously recorded. Recognition is the correct differentiation betweeninformation previously recorded and new information.

Derived from measures over time and in specific contexts, measures ofmemory recall and recognition and memory capacity can be made. These maygenerally fall under the performance category and users may improvememory performance with targeted practice. Recall and recognition areoften improved by semantic similarity among stimuli. Memory span may,likewise, be improved by learning to associate items with one another.The span of time over which items must be remembered may also bemanipulated to alter performance on memory tasks. Distracting tasks, orlack thereof, during the retention span may also heavily influenceperformance.

For long term memory there may be exercises to enhance storage andretrieval, both of specific items and more generally. It may also bepossible to derive measures associated with muscle memory within thecontext of certain physical interactions. Perceptual adaptation andperceptual learning are also candidates for measurement andmanipulation.

Targeting relates to measures of temporal and positional precision inthe user's actions. Referring back to FIG. 1 , targeting may relate tothe precision of the responses of user 102 relative to the position ofobjects in the VE. Targeting is measured as the error between the user'sresponses and an optimal value, in relation to stimuli. The responsecould be a click, touch, gesture, eye movement, pupil response, blink,head movement, body/limb movement, or any other. If the user is expectedto respond precisely in time with some event (as opposed to acting inresponse to that event, leading to a Reaction measure), they may respondtoo early or too late. The variability in the precision of theirresponse yields a Targeting time error measure (usually on the order ofone second or less). Additionally the position of the user's responsesmay have either a consistent bias (mean error) and/or level ofvariability (standard deviation of error) measured in pixels on thescreen or some other physical unit of distance.

In embodiments, the system analyzes data related to user errors,including incorrect choices and deviations made by the user from theideal or an optimum response. Most commonly these may bemisidentification of stimuli, responding at inappropriate times (falsepositive responses), failing to respond at appropriate times (falsenegatives) and inaccuracy of timing or position of responses.Variability in responses or measures of response features may also beindications of error or general inaccuracy or inconsistency.

If a user's targeting score is abnormally low it may be that targets aretoo small or variability of location is too great. For timing ofresponses, more consistent timing of events makes synchronizingresponses easier. This may be in the form of a recurring rhythm or a cuethat occurs at some fixed time before the target event. For position,errors can be reduced by restricting the possible locations of targetsor, in the case of moving targets, using slower speeds. Particularly fortouch interfaces or other contexts where responses may themselvesobscure the target (i.e. finger covering the display), making the targetlarger may improve targeting scores.

Multi-Tracking (M)

Multi-tracking (M) may generally refer to instances in which users aremaking multiple, simultaneous responses and/or are responding tomultiple, simultaneous stimuli. They also include cases where users areperforming more than one concurrent task, and responses to stimulusevents that occur in the periphery (presumably while attention isfocused elsewhere). Combination measures of peripheral detection(detection as a function of eccentricity) and other performance measuresin the context of divided attention may be included.

Multi-tracking (M) may represent the ability of the user to sensemultiple objects at the same time. Divided attention tasks may requireuser to act upon multiple things happening at once. Multi-Trackingmeasures the relative performance of users when interacting with stimulithat are presented in the context of Focused or Divided Attention. Withfocused attention, users generally need to pay attention to one part ofa scene or a limited number of objects or features. In situationsrequiring divided attention, users must monitor multiple areas and runthe risk of missing important events despite vigilance. As with Field ofView, measures of Accuracy and Detection are used to determine a user'sperformance in the different Multi-Tracking contexts.

If a user's Multi-Tracking score is abnormally low it may indicate thatthey are performing poorly with tasks requiring Divided Attention, orexceptionally well with tasks requiring Focused Attention. Therefore,making Divided Attention tasks easier or Focused Attention tasks moredifficult may improve the Multi-Tracking score. In the context ofDivided Attention, reducing the perceptual load by decreasing the numberof objects or areas the user needs to monitor may help. Increasingdurations (object persistence) and slowing speeds in Divided Attentionmay also improve scores.

Fatigue/Endurance (E)

Performance measures may become worse over time due to fatigue. This maybecome evident in reductions in sensitivity (detection), correctdiscrimination, increase in response time and worsening rates ormagnitudes of error. The rate of fatigue (change over time) andmagnitude of fatigue (maximum reduction in performance measures) may betracked for any and all measures. The delay before fatigue onset, aswell as rates of recovery with rest or change in activity, maycharacterize endurance.

Endurance (E) may be related to the ability of user to maintain a highlevel of performance over time. Endurance measures relate to trends ofAccuracy and Detection scores over time. Two measures for Endurance areFatigue and Recovery.

Fatigue is a measure of how much performance decreases within a span oftime. Fatigue is the point at which the performance of user may begin todecline, with measures of a rate of decline and how poor the performancegets. The basic measure of fatigue may be based on the ratio of scoresin the latter half of a span of time compared to the earlier half. Weassume that, given a long enough span of time, scores will decrease overtime as users become fatigued and therefore the ratio will be lessthan 1. A ratio of 1 may indicate no fatigue, and a ratio greater than 1may suggest learning or training effects are improving performance alongwith a lack of fatigue. If a user's Fatigue score is abnormally low thenthey may want to decrease the length of uninterrupted time in which theyengage with the task. Taking longer and/or more frequent breaks mayimprove Fatigue scores. Generally decreasing the difficulty of tasksshould help as well.

Recovery is a measure of performance returning to baseline levelsbetween spans of time, with an assumed period of rest in the interveninginterval. Recovery may relate to using breaks provided to usereffectively to return to optimum performance. The basic measure ofrecovery currently implemented is to compare the ratio of scores in thelatter half of the first of two spans of time to the scores in theearlier half of the second span of time. The spans of time may be chosenwith the intention of the user having had a bit of rest between them. Weassume that, given long enough spans of time to ensure some fatigue isoccurring, scores will be lower before a break compared to after andtherefore the ratio will be less than 1. A ratio of 1 indicates noeffect of taking a break, and a ratio greater than 1 may indicate adecrease in engagement after the break or the presence of fatigueacross, and despite, the break.

If a user's Recovery score is abnormally low, they may want to takelonger breaks. It's possible they are not experiencing sufficientfatigue in order for there to be measurable recovery. Challenging theuser to engage for longer, uninterrupted spans of time may improverecovery scores. Likewise an increase in task difficulty may result ingreater fatigue and more room for recovery.

Detection/Discrimination (D)

Detection/Discrimination (D) may refer to the ability of the user todetect the presence of an object, or to differentiate among multipleobjects. This parameter may depend on the sensitivity of user to variousattributes of the object. Whenever a response event signals awareness ofa stimulus event it may be determined that a user detected thatstimulus. Unconscious processing, perhaps not quite to the level ofawareness, may also be revealed from electrophysiological or otherresponses. Detection can be revealed by responding to the location ofthe stimulus or by a category of response that is congruent with thepresence of that stimulus (e.g. correctly identifying some physicalaspect of the stimulus). The magnitude of a stimulus featureparameter/value necessary for detection may define the user's detectionthreshold. Any feature of a stimulus may be presumed to be used fordetection, however it will only be possible to exclusively attributedetection to a feature if that feature was the only substantial definingcharacteristic of the stimulus or if that stimulus feature appears in agreat variety of stimuli to which users have made responses.

Whenever users correctly identify a stimulus feature parameter/value ormake some choice among multiple alternatives based on one or morestimulus features that interaction may contribute towards a measure ofdiscrimination. In many cases the measure of interest may be howdifferent two things need to be before a user can tell they aredifferent (discrimination threshold). Discrimination measures mayindicate a threshold for sensitivity to certain features, but they mayalso be used to identify category boundaries (e.g. the border betweentwo named colors). Unlike detection measures, discrimination measuresneed not necessarily depend upon responses being correct/incorrect.Discrimination measures may indicate subjective experience instead ofability.

Measures of Detection/Discrimination may be divided into threesubcomponents: measures related to detecting and/or discriminating Color(chromoluminance), Contrast (chromoluminant contrast), and Acuitymeasures based on the smallest features of a stimulus. These afferentproperties, in combination with efferent measures from manual or vocalresponses, eye tracking measures (initiation of pro-saccade, decrease inanti-saccade, sustained fixation and decreased blink response), gazedirection, pupil size, blinks, head tracking measures,electrophysiological and/or autonomously recorded measures, measuresfrom facial pattern recognition and machine learning, and others, asdiscussed above, are used to determine sensitivity. All measures may bebased on a user's ability to detect faintly visible stimuli ordiscriminate nearly identical stimuli. These measures are tied to thedifferent subcomponents based on differences (between detected objectsand their surroundings or between discriminated objects) in theirfeatures. Stimulus objects can differ in more than one feature andtherefore contribute to measures of more than one subcomponent at atime.

Color differences may refer specifically to differences in chromaticityand/or luminance. If a user's Color score is abnormally low, tasks canbe made easier by increasing differences in color. Specific colordeficiencies may lead to poor color scores for specific directions ofcolor differences. Using a greater variety of hues will generally allowspecific deficiencies to have a smaller impact and stabilize scores.

Contrast differs from Color in that contrast refers to the variabilityof chromaticity and/or luminance within some visually defined area,whereas measures relating to Color in this context refer to the meanchromaticity and luminance. If a user's Contrast score is abnormally lowit may be improved by increasing the range of contrast that is shown.Contrast sensitivity varies with spatial frequency, and so increasing ordecreasing spatial frequency (making patterns more fine or coarse,respectively) may also help. Manipulations that improve Color scoreswill also generally improve Contrast scores.

Acuity measures derive from the smallest features users can use todetect and discriminate stimuli. It is related to contrast in thatspatial frequency is also a relevant physical feature for measures ofacuity. If a user's Acuity score is abnormally low it may be thatobjects are generally too small and should be enlarged overall. It mayalso help to increase differences in size, increase contrast anddecrease spatial frequency. More so with Acuity than Color or Contrast,the speed of moving stimuli can be a factor and slowing moving targetsmay help improve Acuity scores.

The above parameters are all based on measuring features. Inembodiments, their patterns may be noted over time. Trends and patternsmay enable predictive analytics and also help personalize the userexperience based on detection capabilities and other VPI/FAMEDcapabilities of the end user.

A great many general states of being may be inferred from the directmeasures discussed. States may be estimated once per session, forcertain segments of time or on a continuous basis, and in response tostimulus events. These may commonly relate to rates of responding orchanges in behavior. FIG. 14 provides a table containing a list ofexemplary metrics for afferent and efferent sources, in accordance withsome embodiments of the present specification. The table illustratesthat an afferent source may result in a stimulus event and feature. Thecombination of afferent source, stimulus events and feature, whencombined further with a response (efferent source), may indicate aresponse event and feature. These combinations may hint at apsychometric measure. In the last column, the table provides adescription for each psychometric measure derived from the variouscombinations.

FIG. 15 is an exemplary flow diagram illustrating an overview of theflow of data from a software application to the SDEP. At 1502, asoftware application that may provide an interface to a user forinteraction. The app may be designed to run on an HMD, or any otherdevice capable of providing a VR/AR/M×R environment for userinteraction. Information collected by the application software may beprovided to a Software Development Kit (SDK) at 1504. The SDK works witha group of software development tools to generate analytics and dataabout use of the application software. At 1506 the data is provided assession data from the SDK to the SDEP. At 1508, session data ispre-processed at the SDEP, which may include organizing and sorting thedata in preparation for analysis. At 1510, stimulus and response datathat has been pre-processed is generated and passed further for analysisand processing. At 1512, data is analyzed and converted to performanceindices or scores or other measures of perceivable information, such asVPI scores. At 1514, the analyzed data is sent back to the SDK and/orapplication software in order to modify, personalize, or customize theuser experience. In embodiments data is passed from 1502, fromapplication software through the chain of analysis, and back to theapplication software non-intrusively, in real time.

FIG. 16 illustrates an exemplary outline 1600 of a pre-processing partof the process flow (1508, FIG. 15 ).

FIG. 17 is an exemplary representation 1700 of the programming languageimplementation of a data processing function responsible for taking inraw data (pre-processed), choosing and implementing the appropriateanalysis, sending and receiving summary measures based on the analysisto temporary and long-term stores for estimates of ‘endurance’ measuresand score normalization, respectively, and computing scores to be sentback to the application for display to the end user. In embodiments, theprogramming language used is Python. The figure shows application ofseveral Python functions to FAMED data in order to derive VPI scores.The figure illustrates color-coded processes for each FAMED function. Inan embodiment, FOV functions are in Red, Accuracy in Green,Multi-Tracking in Purple, Endurance in Orange, and Detection in Blue. Inan embodiment, parallelograms represent variables; rounded rectanglesrepresent functions; elements are color coded for user/session data,which are shown in yellow.

Referring to the figure, contents of a large red outline 1702 representthe processing function (va_process_data), which includes three mainsections—a left section 1704, a middle section 1706 and a right section1708. In an embodiment, left section 1704 takes in raw data and applieseither Accuracy or Detection/Discrimination analysis functions to thedata yielding a single measure summarizing the incoming data. That issent to middle-level functions 1706 for measures of Field of View andMulti-Tracking as well as to an external store. That first externalstore, or cache, returns similar measures from the recent past to beused for measures of Endurance. The output from the middle-levelfunctions 1706 are sent to another external store that accumulatesmeasures in order to estimate central tendency (i.e. arithmetic mean)and variability (i.e. standard deviation) for normalization. Data fromthis second, external store are combined with the present measurementsto be converted into Scores in the right-level section 1708. The figurealso illustrates a small sub-chart 1710 in the lower left of the figureto show the place

Visual Data Packages: Examples of Use

Data generated by the SDEP in accordance with various embodiments of thepresent specification may be used in different forms. In embodiments,data output by the SDEP may be packaged differently for medical use(visual acuity, eye strain, traumatic brain injury, and sports visionperformance), for athletes/sports, and others. For example, applicationsinclude the ability to track the effects of digital eye strain over aperiod of time or to screen for traumatic brain injury in contact sportssuch as football by measuring key areas of the eye-brain connection.

In embodiments, the SDEP allows for advantageously using data generatedfrom technologies such as smart devices, wearables, eye-tracking tools,EEG systems, and virtual reality and augmented reality HMDs.

Performance indices, including VPI, may be different for differentapplications. In an example, detection and accuracy metrics aredifferent for a gaming media vs. media for an advertisement. Someexemplary embodiments of a few applications are described below.

FIG. 18 illustrates an exemplary environment for implementing a centralsystem 1802 that utilizes the SDEP to process psychometric functions andmodel visual behavior and perception based on biomimicry of the userinteraction. In an example, as described below, a user may be presentedwith an interactive electronic media, similar to a game, in which theyare required to ‘pop’ balloons that appear at different locations on ascreen. In this example, system 1802 may utilize psychometric functionsto measure vision psychometrics that are subsequently presented to theuser as FAMED insights. Similarly, there may be other forms ofinteractive media that enables collection of psychometric information inrelationship to visual perception and spatial orientation. FIG. 18illustrates various sensory psychometric data interacting with system1802 and each other to enable processing through SDEP and subsequentmodeling and thereafter support artificial intelligence systems.

More specifically, the present specification describes methods, systemsand software that are provided to train and develop deep learningsystems in order to mimic the human sensory system. In some embodiments,the system may also train and develop deep learning systems that mimichuman facial expressions. In an embodiment, a central systemcommunicates with one or more SDEP systems, and a plurality of autonomicand somatic sensors to collect data that can be used to train learningroutines.

Gaming Applications to Measure User's Vision Performance

In an embodiment, the present specification describes methods, systemsand software that are provided to vision service providers in order togather more detailed data about the function and anatomy of human eyesin response to various stimuli. The detailed data may relate todifferent aspects of a user's vision, and may be compared withcorresponding standard vision parameters, to generate vision performancescores. The scores may be for each different aspect of vision, and/ormay be combined to present an overall vision performance score. Thescore is also presented to the user as a measure of the user's visionperformance. In embodiments, various stimuli is provided to the userthrough a mobile or any other gaming application that may be accessed bythe user.

An exemplary application (hereinafter, referred to as “Sight Kit”), maybe designed to measure the performance of a visual system through aseries of interactive games. While the specification describes featuresof the Sight Kit gaming application, they should be considered asexemplary embodiments only. Alternative embodiments are possible andwill be apparent to those skilled in the art. Alternative embodiment mayinclude variations and/or improvements in one or more of context,sequence, gaming levels, graphical representations, scoring systems,reporting methods, user-interface, and other aspects, in the gamingapplication. The gaming application may report a set of scores to theuser in various categories. These scores can be an indication of how theindividual user performs relative to all users. A weighted average of auser's scores may yield the Vision Performance Index (VPI) which, justlike the component scores, may represent the user's vision performancerelative to a baseline, such as the broader population.

In an embodiment, a user engages with a gaming application in accordancewith embodiments of the present specification. In an example, the gamingapplication is referred to as the Sight Kit application. In anembodiment, the Sight Kit application is presented to the user through amobile platform, such as a smartphone or any other portable electronicdevice including HMDs. In an embodiment, the user is presented with aseries of views through the electronic device, which sequentially enableaccess to a type of stimulation, which could be in the form of one ormore interactive games. In an embodiment, the user is able to securelyaccess the Sight Kit application. In embodiments, a single mobileplatform is used by multiple users to securely access the Sight Kitapplication. Secure access is enabled through a secure authenticationprocess. Following are exemplary views and information presented to theuser through the display of the electronic device, while attempting tosecurely access the Sight Kit application:

FIG. 19 illustrates screenshots 1900 of empty and error screens that mayappear through the sight kit application, in accordance with anembodiment of the present specification.

FIG. 20A illustrates a screenshot 2000A of splash screen that may appearthrough the sight kit application, in accordance with an embodiment ofthe present specification.

FIG. 20B illustrates a screenshot 2000B of home screen that may appearthrough the sight kit application, in accordance with an embodiment ofthe present specification.

In embodiments, the application enables secure access. FIG. 20Cillustrates a series of screenshots 2000C of the login (registration)process including an exemplary registration by a user named ‘Jon Snow’that may appear through the sight kit application, in accordance with anembodiment of the present specification.

FIG. 20D illustrates a screenshot 2000D of a screen with terms andconditions that may appear through the sight kit application, inaccordance with an embodiment of the present specification.

FIG. 20E illustrates a series of screenshots 2000E that may appearthrough the sight kit application in case a user forget their logininformation, in accordance with an embodiment of the presentspecification.

In embodiments, the user is prompted for personal information at thetime of accessing the application for the first time. For example, theuser is prompted for demographic information. In some embodiments, thedemographic information is subsequently utilized to determine a standardor average score for similar demographics, which may be used forcomparison of the user's score.

FIG. 21A illustrates a series of screenshots 2100A of screens thatprompt a user with demographic questions that may appear through thesight kit application, in accordance with an embodiment of the presentspecification.

FIG. 21B illustrates a further series of screenshots 2100B of screensthat prompt a user with demographic questions that may appear throughthe sight kit application, in accordance with an embodiment of thepresent specification.

FIG. 21C illustrates still further series of screenshots 2100C ofscreens that prompt a user with demographic questions that may appearthrough the sight kit application, in accordance with an embodiment ofthe present specification.

FIG. 22 illustrates a series of screenshots 2200 of screens that presenta user with an initial VPI report that may appear through the sight kitapplication, in accordance with an embodiment of the presentspecification. In an embodiment, the initial VPI report is an example ofa set of scores for other users with demographics similar to the actualuser. In another embodiment, the initial VPI report is present to areturning user, and includes previous scores achieved by the user.

FIG. 23 illustrates screenshots 2300 of different screens that mayappear at separate times, prompting a user to select a game to play thatmay appear through the sight kit application, in accordance with anembodiment of the present specification. In embodiments, the userinterface differs based on the past interaction of the user. The usermay be presented with information about the games they have playedpreviously.

In an embodiment, the Sight Kit application is divided into three games.Within games are successive rounds with more or less alteredexperiences.

Game 1: Pop the Balloons

In this round, users may be required to tap in response to theappearance of some visual stimuli (targets) and not others(distractors). This provides data suitable to psychometric curve fittingwhere the proportion of correct discriminations (tapping targets vs. nottapping targets) as a function of color, contrast or acuity differencescan be used to estimate discrimination thresholds (i.e. detectionmeasures, as described above). The game may encourage speedy responsesto specific areas of the display which provides data for Reaction timeand Targeting precision (i.e. Accuracy measures, as described above).The game may have multiple rounds, which may be presented to the user ina sequence. Alternatively, the user may choose to interact with anyround. FIGS. 24A to 24F illustrate various interfaces seen for the gameof ‘Pop the Balloon’.

Round 1

FIG. 24A illustrates a screenshot 2400A of Pop the Balloons Round 1instructions, which may be presented through the sight kit applicationin accordance with an embodiment of the present specification.

FIG. 24B illustrates a screenshot 2400B of Pop the Balloons Round 1game, which may be presented through the sight kit application inaccordance with an embodiment of the present specification.

The first round of Pop the Balloons features balloons rising from thebottom of the screen to the top (‘floating’ into and out of view at thedisplay edges). Some balloons feature a striped pattern while others aresolid, and users may tap the striped balloons while ignoring the solidones (contrast discrimination). The colors used for each balloon may berandom (although alternating stripes in the striped balloons are white).The size of balloons may decrease over time. The changing size mayreflect acuity influence, both in balloon size and spatial frequency ofstripes within balloons. In embodiments, the speed of movement mayincrease over time and the contrast of the striped patterns may decreaseover time. At the beginning of the round, balloons may appear one at atime. Such an appearance may provide and measure focused attention ofthe user. Gradually, more balloons may be presented on the display at atime, requiring tracking of multiple objects at once. Presentingmultiple balloons at the same time may probe divided attention of theuser. An optimal strategy early on might be to look to the middle of thebottom edge of the display to catch balloons as they first appear;therefore the horizontal position of the appearing balloons might bemore or less distant from fixation. This may help determine the userparameters corresponding to field of view.

Round 2

FIG. 24C illustrates a screenshot 2400C of Pop the Balloons Round 2instructions, which may be presented through the sight kit applicationin accordance with an embodiment of the present specification.

FIG. 24D illustrates a screenshot 2400D of Pop the Balloons Round 2game, which may be presented through the sight kit application inaccordance with an embodiment of the present specification.

In this round, balloons do not move, but rather appear very briefly.There is no color or contrast variety, and acuity may be the primarymechanism for discrimination. Users may pop the balloon shapes whileignoring other, similar shapes. The more similar the shape is to aballoon, the harder it may be to discriminate leading to false positiveresponses. Variation in color differences from the background may beadded as an additional source of color discrimination measures.

In this game, Reaction times and Targeting precision are may be a majorcomponent for measuring Accuracy. An optimal strategy might be to fixateon the center of the display giving rise to a Field of View component.Objects may appear one at a time, with gaps of time in between, negatingthe possibility of a Multi-Tracking measure.

Round 3

FIG. 24E illustrates a screenshot 2400E of Pop the Balloons Round 3instructions, which may be presented through the sight kit applicationin accordance with an embodiment of the present specification.

FIG. 24F illustrates a screenshot 2400F of Pop the Balloons Round 3game, which may be presented through the sight kit application inaccordance with an embodiment of the present specification.

In the third round, which may also be a final round, balloons mayneither move nor appear briefly; instead difficulty may be increased byintroducing a feature conjunction search task with increasing set size.Users may find the matching color/shape combination requiring color andacuity discrimination (an indication of Detection). Reaction time may bean important characteristic, with Targeting precision of reducedinterest given the static and persistent nature of the stimuli (anindication of Accuracy). Field of View may also be somewhat indicated,although targets randomly placed towards the center may be found fasteron average than when targets are towards the edges of the balloonclusters. Multi-Tracking may have a significant impact here, dependingupon whether users employ serial or parallel processing of visualstimuli; this may be revealed later by the dependency, or lack thereof,of set size on reaction time (Hick's law).

Game 2: Picture Perfect

In this game, an image may be displayed to the user along with itsdistorted version. The user may be provided with tools to vary displayparameters of the distorted image in order to match it to the originalimage. In an embodiment, the display parameters may include acombination of one or more of color, sharpness, and size, among other.Once the user confirms completing the task, results may be presented tothe user by comparing the user's selections and the correct selections.In an embodiment, greater the proximity of the user's selection to thecorrect selection, the greater is the vision performance of the user.The Picture Perfect game may not require a fast reaction, although usersmay be encouraged to work fast (for example, the number of settings madein a fixed period of time may be used to generate a Reaction score). Inan embodiment, the game consists of multiple rounds. FIGS. 25A and 25Billustrate a series of screenshots 2500A and 2500B respectively, ofPicture Perfect Round 1 game, which may be presented through the sightkit application in accordance with an embodiment of the presentspecification. In some embodiment, sliders are provided to the user tovary different parameters in order to correct a distorted image. Inother embodiments, other graphical, numerical, or any other, tools canbe provided for this purpose.

FIGS. 25C, 25D, and 25E illustrate a series of screenshots 2500C, 2500D,and 2500E respectively, of Picture Perfect Round 2 game, which may bepresented through the sight kit application in accordance with anembodiment of the present specification. The advanced round may presentthe user with the original image and the distorted image at separatetimes, and not simultaneously. The user is then required to correct thedistorted image by recalling the original image from their memory.

FIG. 25F illustrates a screenshot 2500F of an exemplary after gamereport for a user, which may be presented through the sight kitapplication in accordance with an embodiment of the presentspecification.

The Picture Perfect game may enable partial indication of visionparameters related to Field of View and Multi-Tracking, because usersmay freely sample the visual scene without restriction. Depending onwhich sliders are available to users in a given round, various measuresof discrimination (Detection) may be made. Scores may be inverselyproportional to the magnitude of error between the correct level of eachadjustment slider and the user's settings. ‘Color’, ‘Hue’ and‘Saturation’ adjustments may contribute to Color measurements. ‘Size’and ‘Sharpness’ adjustments may contribute to Acuity measurements.

Game 3: Shape Remix/Memory Match

Instructions to interact with the game may be optionally provided to theuser before starting the game. In an embodiment, the user is presentedwith an original image including multiple elements. The task for theuser is to edit the elements in an alternate image, in order to matchthe element and their layout as previously shown in the original image.In some embodiments, the user is provided with tools that enable varyingdifferent characteristics of each element. For example, the user is ableto vary the hue, saturation, contrast, sharpness, size, or any other,parameter separately for each element. Once the user confirms completingthe task, the result may be presented by displaying the original imageadjacent to the image recreated by the user. Additionally, numericalscores and verbal reactions may be presented to the user.

FIGS. 26A, 26B, and 26C, illustrate similar set of screenshots 2600A,2600B, and 2600C respectively, for ‘Shape Remix’ game, its instructions,and after game report, which may be presented through the sight kitapplication in accordance with an embodiment of the presentspecification.

Game 4: Speed Pop

Instructions to interact with the game may be optionally provided to theuser before starting the game. In an embodiment, the user is presentedwith streaming shapes and images, which includes balloons and anassortment of other shapes. The task for the user is to tap any balloonwhile avoiding tapping any other shape. In an embodiment, both theballoons and assortment of other shapes are colored the same.Additionally, numerical scores and verbal reactions may be presented tothe user.

Game 5: Match Pop

Instructions to interact with the game may be optionally provided to theuser before starting the game. In an embodiment, the user is presentedwith an example object having a shape and a color. The objective of thegame is for the user to tap the balloon that includes an object thatmatches the example object provided, with respect to both shape andcolor. Additionally, numerical scores and verbal reactions may bepresented to the user.

Game 6: Star Catch

Instructions to interact with the game may be optionally provided to theuser before starting the game. In an embodiment, the user is expected tonavigate a ship to collect target shapes, where the target shapes may bedefined for or presented to the user beforehand. Additionally, numericalscores and verbal reactions may be presented to the user.

In an embodiment, an after game report is generated after each game isplayed and provides a user with their overall VPI as well as how eachFAMED component was affected by their performance in the game. Eachreport may also show how the user performed compared to their age group.In addition, each report may provide an option for the user to learnmore about a VPI in depth. A fun fact may also be presented alongsidethe report, in addition to a directory and/or map of local eyespecialists.

FIG. 27 illustrates screenshots 2700 of VPI game reports after playingdifferent games that may appear through the sight kit application, inaccordance with an embodiment of the present specification.

FIG. 28 illustrates some screenshots 2800 that may appear based on theuser's VPI report, where the screens suggest doctors and/or eye-carepractitioners, in accordance with an embodiment of the presentspecification.

FIG. 29 illustrates some screenshots 2900 of the screens that present auser's profile that may appear through the sight kit application, inaccordance with an embodiment of the present specification. Eachscreenshot present the profile of the user over a different time spanand/or at different points of time. The user may select to view detailsof the VPI through their profile.

FIG. 30A illustrates some screenshots 3000A of the VPI breakdown thatmay appear through the sight kit application, in accordance with anembodiment of the present specification.

FIG. 30B illustrates some screenshots 3000B of the VPI breakdown thatprovide details about each FAMED parameter, through the sight kitapplication in accordance with an embodiment of the presentspecification.

FIG. 30C illustrates some screenshots 3000C of the VPI breakdown thatprovide details of parameters within each FAMED parameter, through thesight kit application in accordance with an embodiment of the presentspecification.

FIG. 30D illustrates some screenshots 3000D of the VPI breakdown thatprovide further details of parameters within each FAMED parameter,through the sight kit application in accordance with an embodiment ofthe present specification.

FIG. 31 illustrates screenshots 3100 for ‘Settings’ and related optionswithin ‘Settings’, which may be presented through the sight kitapplication in accordance with an embodiment of the presentspecification.

From the perspective of the VPI and its components, the Shape Remix gamemay be similar in contributing data largely for Detection measures.Though there may be differences in the nature of the effect of color,size, position, and contrast on performance. User performance may or maynot be equivalent on the two games (Picture Perfect and Shape Remix),while the two games may not be considered redundant. In embodiments,values from the Shape Remix game may be complimentary with that of thePicture Perfect game.

VPI may be determined for the user for each level, each game, and/or forall games played by the user. In some embodiments, a comprehensive VPIreport is presented to the user. The comprehensive report may be basedon data and score identified through user's interaction with all thegames. In some embodiments, the report additionally takes in toconsideration different score over a time period over which the user mayhave interacted with the games. The system may provide additionaldisplay options to view an overall VPI score, in addition to VPI scoresfrom each game.

In some embodiments, the system offers the user to repeat interactionwith one or more games, until the user is satisfied with the VPI scores.In embodiments, Sight Kit and the resultant VPI score or report may beused to increase awareness of the user's visual system, provideinformation and develop understanding, and to enable tracking of visionperformance over time. Sight Kit may provide a general overview ofvision performance and highlight areas for potential improvement.

In an embodiment, Sight Kit provides continuous feedback and uses avariety of stimuli and responses to form a comprehensive picture, thuspotentially providing a vast trove of vision data. The VPI scores enablethe user to be more aware of their vision and to monitor their visionperformance over time. Sight Kit application may measure aspects ofusers overall vision, inform them of where they may be not performing attheir best and provide tips to help maintain their vision at a highlevel. VPI is designed to give user scores related to specific areas oftheir vision.

The VPI comprises data measures that are indicative of five components:Field of View, Accuracy, Multi-Tracking, Endurance and Detection(F.A.M.E.D.), which were introduced and described in previous sectionsof the present specification. For embodiments of the sight kitapplication, Detection and Accuracy may be considered to be primarymeasures representing estimates of the user's visual system performanceparameters like contrast sensitivity or reaction time. Field of View,Multi-Tracking and Endurance may be considered to be secondary measuresthat compare primary measures in different contexts like parts of thevisual field, focused or divided attention or prolonged engagement.

Each component is further divided into subcomponents. Within the VPIsystem, each subcomponent is scored, a weighted average of subcomponentscores and other measures is used to generate a component score andfinally a weighted average of component scores yields the VisionPerformance Index (VPI). Any given experience in the Sight Kitapplication may only test some of these components, and only bycompleting all of them can a full picture of the VPI be made. Thosesubcomponent elements are further described below.

Field of View

Field of View may be a derived, secondary measure. In the VPI systemthis means that the scores are based on comparing primary measures basedon where stimuli appear in the visual field. Certain experiences withinthe Sight Kit application imply a strategy of fixating on the center ofthe screen (or bottom-center) and monitoring the surrounding displayarea for targets (i.e. the first and second rounds of Pop the Balloons).In these contexts we can label stimuli as Central or Peripheral based ontheir position relative to the presumed area of fixation.

This system may be verified with eye tracking. The field of view scoresfrom a mobile application using sight kit may be somewhat related toperimetry testing of central vision in the clinic.

Central Vision

Central vision scores are based on Detection and Accuracy measures wherethe stimulus is assumed to be near the central visual field (at stimulusonset). This may be specifically relevant for those occasions whereusers must make a speeded response (Pop the Balloons). The relevance mayreduce for the final round of ‘Pop the Balloons’.

Peripheral Vision

Peripheral vision scores are based on Detection and Accuracy measureswhere the stimulus is assumed to be at the edge of the display. Forexample, peripheral vision scores are determined where the stimulus isroughly within the outer left and right/top and bottom thirds of thedisplay. Peripheral stimulus onset may be assumed in those contextswhere an optimal strategy involves fixating at the center (orbottom-center) of the display, such as the first and second rounds of‘Pop the Balloons’.

Accuracy

Within the VPI system, accuracy is split into two components: onetemporal and one spatial. In case of spatial accuracy, users may berated on their ability to precisely position their responses (i.e.hitting the bullseye), and it is assumed that response positionsrelative to the intended target will fall in a normal distribution. Inthe temporal case it is the time users take to respond that is measured,and it is assumed that the time taken to respond to a stimulus after itappears will generally follow a log-normal distribution. In analternative embodiment, Sight Kit may also include a second temporalmodel where the time of response is normally distributed around thestimulus onset time (with responses occurring both before and afterstimulus onset) for cases where users can anticipate the appearance of astimulus and respond in synchrony.

Reaction

Reaction times are generated by subtracting the stimulus onset time(when it appears, usually instantly in Sight Kit) from the user'sresponse time. These time spans may be distributed in a log-normalfashion with a characteristic mode and full-width-half-maximum. Reactionscores may be generally based on the mode of a distribution fit to thedata. Variations due to Hick's Law may be present, but may not directlyinfluence a user's score as reaction times are fit without regard to setsize (most relevant for the third round of Pop the Balloons). Reactiontimes may be most relevant for the Pop the Balloons game.

Targeting

Targeting precision of response position may be based on the distance(measured in pixels) between the position of the user's response (e.g.tapping to pop a balloon) and the center of the stimulus to which theuser is responding and is a basic measure of eye-hand coordination(loosely related to the Compensatory Tracking Task). This measure isfrom the Pop the Balloons' game, although manual dexterity may minimallyinfluence the other games. Targeting precision and reaction time mayhave an inverse relationship to each other in that the more careful auser is about their aim the slower their responses may be. This effectmay average out when calculating a user's overall Accuracy score.

Multi-Tracking

Multi-Tracking may be based on comparing primary measures in the contextof either focused or divided attention. Here, attentional demands areproportional to both the number of concurrent tasks and the number ofconcurrent stimuli.

Focused Attention

Focused attention may be considered to be the state associated withDetection or Accuracy measures where users have only one task to performand one stimulus to consider at any given time. This is generally thecase for the ‘Picture Perfect’ and ‘Shape Remix’ games as users are freeto process stimulus features serially. This may also be applicable forthe beginning of the first round of Pop the Balloons as well as for theentire second round.

Divided Attention

Divided attention is assigned to those primary measures when more thanone stimulus (targets or distractors) is present at once and a speededresponse requires parallel processing of stimuli. The first and thirdrounds of Pop the Balloons' fit this description in that users may makespeeded responses with multiple stimuli present.

Endurance

Endurance measures may relate to how users perform on primary measuresover time. Given prolonged engagement it is assumed that performancewill begin to decline. After a rest, this loss may be recovered andperformance may be back to normal. This assumption relies on usersplaying the games to the point of fatigue, which may not be reasonable.Interest may fail before performance has a chance to do so. It ispossible to add a consideration for time of day, as this is of interestin the analytics dashboard. The application may also consider thecurrent and previous play sessions to generate Endurance related scoresrelevant to how the user is performing in real time.

Endurance scores are relevant for all experiences within the Sight Kitapplication. Generating endurance scores does, however, require someminimal duration of engagement with the different games in order tocompare present and past data. The relevance of endurance scores dependsnot on what is played but rather the accumulation of play time.

Fatigue

Fatigue relates scores based on the primary measures from the early andlater halves of an ongoing gaming session. Scores are higher if usersmaintain their level of performance (or even improve over time as withpractice) and lower if users' performance begins to slip.

Recovery

Recovery relates scores based on the primary measures from the laterhalf of the last session to the first half of the current session. Iffatigue, as described above, has resulted in lower scores at the end ofthe last session, and if a rest has allowed performance to return tobaseline, recovery scores may be higher. If the rest was insufficientand there is little or no recovery from fatigue, recovery scores may belower.

Detection

Detection measures broadly encompass both measures of sensitivity(whether or not a stimulus is seen) and measures of discrimination(choosing from among similar stimuli). Sight kit may enable successfuldiscrimination of stimuli as a function of color, contrast, or acuity.In this case, a sigmoid function may be fit by Bayesian estimation tothe data to derive a discrimination threshold. The games that interactwith the user probe errors in matching performed by method ofadjustment. In this case, the error in the user's settings may be takenas a direct measure of discrimination threshold.

Color

Color performance in Sight Kit may be based on responses with regard tothe chromatic distance between stimuli being discriminated. Theapplication may create a balanced distribution of color discriminationdirections and is diagnostic of specific color deficiencies. User'scolor performance may be based on how fine a discrimination they canmake, based on color compared to other users. Users with a marked colordeficiency, such as users suffering from dichromacy, may see notablylower scores. Users with slight deficiencies, such as those sufferingfrom anomalous trichromacy, may see lower scores but perhaps within the‘normal’ range.

Color scores may be generated by the first and third rounds of Pop theBalloons' game and selected rounds within ‘Picture Perfect’ and ‘ShapeRemix’ games where users are asked to adjust the color (hue) orsaturation of images or shapes.

Contrast

Contrast performance in Sight Kit may be based on discrimination betweensimple patterns with little or no contrast (Pop the Balloons), simpleshapes (Shape Remix) and complex photographic images (Picture Perfect).The discrimination in the first round of Pop the Balloons' may be, insome ways, similar to the ‘Vision Contrast Test System’.

Acuity

Acuity performance in Sight Kit may be based on discriminating brieflypresented shapes, speeded response to complex shapes with a mild acuitycomponent, size/alignment (akin to Vernier acuity) and matching blurlevel. Acuity is relevant for all rounds of ‘Pop the Balloons’, to oneextent or another, and to ‘Picture Perfect’ and ‘Shape Remix’ roundswith position, size or sharpness adjustments.

Scoring

The primary measures (Detection and Accuracy) may represent a variety ofpsychophysical measures tied to physical properties of stimuli andresponses, and the secondary measures (Field of View, Multi-Tracking andEndurance) may be largely relational. In order to compare measuresacross very different experiences, and to combine the variouscomponents, a normalizing procedure may be adopted to generate scores.In embodiment, the normalization takes place separately for each uniquecontext, where a context may be considered for each measure for eachround of each game.

Measures may generally be distributed normally among the generalpopulation. Internally, estimates of central tendency (arithmetic mean)and variability (standard deviation) may be made for each measure ineach context based on all of the accumulated data from all users. Insome embodiment, these values are used to convert a user's measure underevaluation to a score based on a mean of ½ and a standard deviation of ⅙(yielding a distribution falling largely between 0 and 1). In someembodiments, these scores are multiplied by a constant to give a largervalue for the score. In an embodiment, a constant of 20 is used. Inembodiments, the scoring system is designed to stay self-consistent asdata are accumulated and it will tell users how they perform relative toall users.

The various VPI components may have some differences from the otherconventional methods of measuring vision performance, especially in thecontext of the few experiences presented by the Sight Kit application.

The Field of View score may not reveal the location or size of a scotomaor field of neglect. Accuracy scores may be influenced by all aspects ofvision performance as well as many factors outside of vision. There maybe a level of individual variability among users in this regardespecially as it will be tied to an individual's affinity for electronicgames. FIG. 32 provides a table 3200 to illustrate exemplary experiencesof different VPI parameters from the different games and rounds.

In some embodiments, round one of Pop the Balloons' game may generatedata related to some of FAMED parameters. The data from this experiencemay inform scores for Detection based on color, contrast and acuity;Accuracy based on reaction and targeting; Field of View andMulti-Tracking with the potential for Endurance given enough play time.The value of this particular experience may primarily be in Accuracyscores, Detection scores based on contrast and acuity and Multi-Trackingscores.

The data from experience with round two of Pop the Balloons' game mayinform scores for Detection based on acuity, Accuracy based on reactionand targeting and Field of View. The value here may be primarily inAccuracy scores and Field of View, with some value in Detection byacuity.

The data from experience with round three of Pop the Balloons' game mayinform scores of Detection based on Acuity and Color and Accuracy basedon Reaction and Targeting. The primary value is in the Detection andReaction measures.

The data from experience with ‘Picture Perfect’ game may inform scoresof Detection based on Color, Contrast and/or Acuity.

The above examples are merely illustrative of the many applications ofthe system of present invention. Although only a few embodiments of thepresent invention have been described herein, it should be understoodthat the present invention might be embodied in many other specificforms without departing from the spirit or scope of the invention.Therefore, the present examples and embodiments are to be considered asillustrative and not restrictive, and the invention may be modifiedwithin the scope of the appended claims.

We claim:
 1. A method of assessing a vision performance of a patientusing a computing device programmed to execute a plurality ofprogrammatic instructions, comprising: presenting, via the computingdevice, a first set of visual and/or auditory stimuli; monitoring afirst plurality of reactions of the patient using at least one of thecomputing device and a separate hardware device; presenting, via thecomputing device, a second set of visual and/or auditory stimuli;monitoring a second plurality of reactions of the patient using at leastone of the computing device and a separate hardware device; presenting,via a display on the computing device, a third plurality of visualand/or auditory stimuli that appear concurrently with said secondplurality of visual and/or auditory stimuli, wherein the third pluralityof visual and/or auditory stimuli are different than the secondplurality of visual and/or auditory stimuli; and based upon said firstplurality of reactions and second plurality of reactions, determiningquantitative values representative of at least three of the patient'sfield of view, visual accuracy, ability of the patient to track multiplestimuli, visual endurance or visual detection.
 2. The method of claim 1further comprising generating a single vision performance valuerepresentative of an aggregation of the field of view, the visualaccuracy, the ability of the patient to track multiple stimuli, thevisual endurance and the visual detection.
 3. The method of claim 1wherein the first plurality of reactions comprises at least one of rapidscanning data, saccadic movement data, blink rate data, fixation data,pupillary diameter data, and palpebral fissure distance data.
 4. Themethod of claim 1 wherein the second plurality of reactions comprises atleast one of rapid scanning data, saccadic movement data, fixation data,blink rate data, pupillary diameter data, speed of head movement data,direction of head movement data, heart rate data, motor reaction timedata, smooth pursuit data, palpebral fissure distance data, degree andrate of brain wave activity data, and degree of convergence data.
 5. Themethod of claim 1 wherein the hardware device comprises at least one ofa camera configured to acquire eye movement data, a sensor configured todetect a rate and/or direction of head movement, a sensor configured todetect a heart rate, and an EEG sensor to detect brain waves.
 6. Themethod of claim 1 wherein the quantitative values representative of thepatient's field of view comprises data representative of a quality ofthe patient's central vision and data representative of a quality of thepatient's peripheral vision.
 7. The method of claim 1 wherein thequantitative values representative of the patient's visual accuracycomprises data representative of a quality of the patient's reactiontime to said first set of visual and/or auditory stimuli.
 8. The methodof claim 1 wherein the quantitative values representative of thepatient's visual accuracy comprises data representative of a quality ofthe patient's precise targeting of said first set of visual stimuli andwherein said quality of the patient's precise targeting of said firstset of visual stimuli is based on a position of the patient's physicalresponse relative to a position of the first set of visual stimuli. 9.The method of claim 1 wherein the quantitative values representative ofthe patient's ability of the patient to track multiple stimuli comprisesdata representative of a quality of the patient's ability tosimultaneous track multiple elements in the second set of visualstimuli.
 10. The method of claim 1 wherein the quantitative valuesrepresentative of the patient's visual endurance comprises datarepresentative of a decrease in the patient's reaction time over aduration of presenting the first set of visual and/or auditory stimuli.11. The method of claim 1 wherein the quantitative values representativeof the patient's visual endurance comprises data representative of animprovement in the patient's reaction time over a duration of presentingthe second set of visual and/or auditory stimuli after a rest period.12. The method of claim 1 wherein the quantitative values representativeof the patient's visual detection comprises data representative of towhat extent the patient sees the first set of visual stimuli.
 13. Themethod of claim 1 wherein the quantitative values representative of thepatient's visual detection comprises data representative of to whatextent the patient can discriminate between similar colored, contrast,or shaped objects in the first set of visual stimuli.
 14. A method ofassessing a vision performance of a patient using a computing deviceprogrammed to execute a plurality of programmatic instructions,comprising: presenting, via a display on the computing device, a firstset of visual stimuli, wherein the first set of visual stimuli comprisesa first plurality of visual elements that move from a peripheral visionof the patient to a central vision of the patient; monitoring a firstplurality of reactions of the patient using at least one of thecomputing device and a separate hardware device; presenting, via adisplay on the computing device, a second set of visual stimuli, whereinthe second set of visual stimuli comprises a second plurality of visualelements that appear and disappear upon the patient physically touchingsaid second plurality of visual elements; monitoring a second pluralityof reactions of the patient using at least one of the computing deviceand said separate hardware device; presenting, via a display on thecomputing device, a third plurality of visual elements that appearconcurrently with said second plurality of visual elements, wherein thethird plurality of visual elements are different than the secondplurality of visual elements; and based upon said first plurality ofreactions and second plurality of reactions, determining quantitativevalues representative of at least three of the patient's field of view,visual accuracy, ability of the patient to track multiple stimuli,visual endurance or visual detection.
 15. The method of claim 14 whereinat least a portion of the first plurality of visual elements have sizesthat decrease over time.
 16. The method of claim 14 wherein at least aportion of the first plurality of visual elements have a speed ofmovement that increases over time.
 17. The method of claim 14 wherein,over time, more of the first plurality of visual elements simultaneouslyappear on said computing device.
 18. The method of claim 14 wherein ifthe patient physically touches any of said third plurality of visualelements, the quantitative value representative of the patient's visualaccuracy is decreased.
 19. The method of claim 14 further comprisingpresenting, via a display on the computing device, a third set of visualstimuli, wherein the third set of visual stimuli comprises a fourthplurality of visual elements; monitoring a third plurality of reactionsof the patient using at least one of the computing device and saidseparate hardware device; and based upon said first plurality ofreactions, second plurality of reactions, and third plurality ofreactions determining quantitative values representative of thepatient's field of view, visual accuracy, ability of the patient totrack multiple stimuli, visual endurance and visual detection.
 20. Themethod of claim 19 wherein the patient is instructed to identify one ofthe fourth plurality of visual elements having a specific combination ofcolor, contrast, and/or shape.
 21. The method of claim 1 wherein if thepatient physically touches any of said third plurality of visualelements, the quantitative value representative of the patient's visualaccuracy is decreased.